```{r setup}
#| warning: false
#| message: false
knitr::opts_knit$set(root.dir = rprojroot::find_rstudio_root_file())
```Complex passing py-R
Old way of giving path to venv, new way is to set it in .Rprofile
```{r}
# reticulate::use_virtualenv(file.path('RpyEnvs', 'pytesting', '.venv'), required = TRUE)
``````{r}
library(reticulate)
```The issue
I previously did some very basic playing with using python and R in the same quarto doc. I immediately ran into issues though when I tried to use it in a real project because I was no longer passing just simple objects back and forth.
The specific issue I ran into was that it was straighforward to pass a pandas dataframe, but if one of the columns has dates, they come in as a list of python environments containing the date objects. They can be parsed, but it takes forever. So the solution is to turn them into something else in python, pass the pandas, and re-date them.
General problem
The real issue here is when we bring a df into R from python and it has a list column with ‘environment’ in it. Then the conversion hasn’t really happened, and doing that conversion post-hoc has to convert every single environment, which is all rows. And that takes forever. It’s not necessarily dates (and as we can see below, sometimes they do work just fine). So, if that happens, rather than taking an eternity to purrr or otherwise go through the column to translate, just put it in something easier in py, bring it over, and put it back.
I do wish I had a better idea about why it happened. I’m sure as it happens more I’ll start to figure it out.
It’s possible this has been fixed in more recent {reticulate}, and that’s why I can’t reproduce? https://github.com/rstudio/reticulate/pull/1266. No, I’m still getting the error where I was before. Still struggling to replicate it, but it does happen in the original case.
A demonstration (sort of)
Say we have a pandas dataframe with a few columns of simple types (numeric, character) and 1000 rows
```{python}
import pandas as pd
import random
import string
randnums = [random.gauss(0, 1) for _ in range(1000)]
allchars = list(string.ascii_lowercase) + list(string.ascii_uppercase)
randchars = [random.choice(allchars) for _ in range(1000)]
simpledf = pd.DataFrame({'rand_nums': randnums, 'rand_chars': randchars}, columns=['rand_nums', 'rand_chars'], index=range(1000))
simpledf
``` rand_nums rand_chars
0 0.133291 H
1 -0.614250 w
2 -0.620462 N
3 -2.493694 N
4 0.805436 n
.. ... ...
995 -1.994659 S
996 0.460565 r
997 0.701467 y
998 1.222810 K
999 1.423616 l
[1000 rows x 2 columns]Now, we can get that into R without too much fuss using py$.
```{r}
rsimple <- py$simpledf
rsimple
``` rand_nums rand_chars
1 0.133290969 H
2 -0.614249581 w
3 -0.620462491 N
4 -2.493693901 N
5 0.805435503 n
6 2.956532749 d
7 -0.269791645 d
8 0.623760030 I
9 0.434931601 j
10 -0.196023029 U
11 2.160265597 s
12 -0.337976341 i
13 -0.074536375 t
14 0.222722866 j
15 0.293960431 g
16 1.098460288 V
17 1.130655504 F
18 0.942662301 L
19 0.038805967 p
20 0.768540675 H
21 -0.738591424 q
22 -0.422585201 A
23 0.945387050 u
24 -0.711961711 S
25 0.663938931 A
26 -0.892279703 D
27 -0.153834699 k
28 -1.009738832 F
29 -0.456634303 J
30 0.325540164 n
31 -1.081435122 P
32 -0.879610895 h
33 -0.107386570 j
34 1.718037769 a
35 1.552609666 f
36 0.522724384 M
37 -0.081958625 h
38 1.321904381 v
39 2.198275363 i
40 1.714438261 O
41 0.048707125 m
42 -0.198145202 A
43 -0.803614480 f
44 -0.055894093 M
45 0.362882326 P
46 -1.846538635 I
47 -1.286636258 j
48 -1.839522128 e
49 0.503038215 X
50 0.887794938 O
51 -0.621214728 D
52 0.625089781 s
53 0.598281571 U
54 0.588028803 E
55 -1.115054378 I
56 1.599606587 P
57 -0.441912383 j
58 -1.301741393 d
59 -0.213642691 g
60 -1.327684241 z
61 0.545201577 W
62 -0.536702669 U
63 1.846664427 P
64 0.761506287 L
65 0.248476075 M
66 0.052190133 z
67 -0.138237629 v
68 0.827792601 t
69 0.005779370 M
70 0.626125878 a
71 0.707466575 t
72 -0.545022812 W
73 1.005888056 d
74 1.022683770 a
75 0.042368768 R
76 -0.890763798 F
77 0.384980444 y
78 -0.467631535 X
79 0.658856630 n
80 -1.044855472 a
81 2.068693297 j
82 -0.552501530 U
83 1.016135162 I
84 -1.131892101 K
85 0.073397538 N
86 0.573699992 f
87 -0.365487317 v
88 -0.028719842 o
89 -0.516597286 G
90 1.257439274 n
91 -0.842566975 x
92 -0.552633968 x
93 -0.419609772 J
94 0.737165147 R
95 0.926204507 R
96 -1.703984053 B
97 0.193253834 o
98 -0.483670319 l
99 -0.220242500 j
100 0.162833763 F
101 -0.821256855 u
102 -0.981962735 C
103 1.502564938 M
104 0.956338467 V
105 1.923728670 b
106 -0.774926316 L
107 1.355735905 p
108 0.220879712 S
109 -0.045146286 t
110 -0.239706416 r
111 1.116304978 F
112 -0.911247570 B
113 -0.377103048 x
114 0.752145352 y
115 -1.121208842 N
116 -0.628543841 r
117 0.684114233 M
118 -0.091952033 G
119 -1.801924421 g
120 0.579427843 e
121 -0.674565416 b
122 -0.661157970 p
123 -0.083969362 L
124 -1.626475583 P
125 -0.818741031 R
126 0.380291716 C
127 -0.129153769 T
128 -2.117605781 C
129 -1.181478925 b
130 0.835466899 L
131 0.972300450 n
132 -0.746167929 C
133 -2.474815876 P
134 0.449600179 M
135 0.388629020 c
136 1.609919724 L
137 -0.996391203 F
138 -0.546480822 P
139 0.053716663 R
140 1.099594386 A
141 -0.552606278 i
142 2.090444549 f
143 1.072774169 j
144 1.508252297 F
145 1.391979642 G
146 -0.205902083 U
147 -0.215596129 A
148 0.121290402 C
149 -0.033819258 u
150 1.258824146 e
151 0.941587394 z
152 -0.506882475 n
153 -0.307031047 S
154 -0.842593360 O
155 -1.045992029 N
156 0.100648368 T
157 -0.620450835 L
158 1.800641216 F
159 -1.277398279 F
160 -0.842739207 r
161 -0.945952392 j
162 -0.018004535 X
163 1.020687016 t
164 0.251196202 m
165 -2.494604787 V
166 -1.006140104 I
167 -0.704367279 a
168 -0.172325666 i
169 0.750710668 s
170 -1.215626673 x
171 0.436325530 A
172 0.234159106 I
173 -0.269909635 V
174 -1.196639509 d
175 0.477298458 G
176 -0.511461692 s
177 0.522874180 R
178 -0.054055863 E
179 -0.372422070 S
180 1.180761839 X
181 0.379987227 D
182 -0.102005828 u
183 0.782634674 t
184 -1.358987326 q
185 2.417957180 v
186 -0.121614405 V
187 -0.523882906 D
188 1.557816440 H
189 -1.086155215 c
190 -1.431025239 e
191 0.472836128 I
192 -0.317541462 x
193 -0.582328887 B
194 2.312490390 i
195 -0.541847947 Y
196 -0.344131888 m
197 -0.913184109 s
198 -1.477406998 z
199 1.961895996 y
200 0.907502683 s
201 -0.051012503 b
202 -0.195251315 F
203 -1.358216826 Z
204 1.789420101 R
205 0.313445902 P
206 -0.799589288 u
207 -0.485112747 W
208 -0.688425153 B
209 0.067223022 l
210 -0.522933397 X
211 -0.929661696 R
212 -0.007888543 B
213 -0.480354971 N
214 2.179086603 a
215 0.672427749 E
216 -0.172343886 A
217 0.686980601 j
218 -0.108323679 k
219 0.304086289 C
220 -0.306608409 K
221 -0.331369481 f
222 1.547642181 E
223 -1.651367646 L
224 -0.798884909 E
225 -0.254334830 p
226 0.500720453 W
227 -0.134019815 D
228 1.125804197 h
229 2.582999116 o
230 0.149131981 Y
231 0.378214574 J
232 1.005898181 Y
233 -1.284547386 l
234 -0.845913636 C
235 -0.428644899 x
236 1.149137048 v
237 -0.509882524 k
238 0.987589439 v
239 0.754700185 w
240 0.140052524 b
241 0.843921087 B
242 -0.065536895 K
243 1.379837637 t
244 -1.399404832 e
245 -1.089223769 n
246 -1.687449580 Y
247 -0.414218580 F
248 0.727049082 d
249 -0.624120630 V
250 0.900054050 P
251 -0.913343944 I
252 0.720818674 l
253 -0.145136897 W
254 -0.117257773 f
255 1.436712608 G
256 0.655844310 w
257 -0.223566120 i
258 -0.205955794 Q
259 -0.293414937 c
260 0.669391459 d
261 -0.544510124 V
262 -0.305366427 K
263 0.060177280 d
264 0.797992895 V
265 -1.894621117 s
266 1.056077782 l
267 -0.335771332 W
268 1.625757230 A
269 0.151245252 f
270 -0.800585955 b
271 0.466719815 a
272 0.620069634 R
273 -2.015213534 X
274 0.396395023 t
275 -1.161510612 j
276 -0.155535506 E
277 0.536285651 v
278 1.212486004 g
279 0.499729205 M
280 0.364111686 o
281 -0.585659414 L
282 0.507073862 b
283 1.080988323 v
284 -0.587718031 d
285 0.573178333 L
286 -0.448026109 s
287 0.018277121 d
288 -0.653194709 J
289 1.984109534 R
290 -0.120536238 s
291 -0.009666630 h
292 -0.270368526 X
293 -0.158613825 D
294 -0.820627611 e
295 0.821071310 z
296 1.798171340 R
297 0.815738298 n
298 -0.517867131 h
299 0.597862211 I
300 0.111636782 A
301 0.726193542 q
302 0.693289890 Q
303 0.798415248 X
304 0.935584710 W
305 0.089466460 N
306 -0.952748507 i
307 0.067181781 f
308 0.384723396 H
309 -0.787840230 j
310 0.918874737 d
311 -1.551456590 n
312 -0.105096848 y
313 -0.995997658 P
314 -1.142917658 H
315 -2.084868616 K
316 -0.562559045 g
317 -0.367899620 t
318 0.958188641 N
319 -0.414075729 n
320 0.247033226 S
321 -0.408690635 N
322 0.355066414 b
323 1.181881647 B
324 -1.942317961 p
325 0.687417940 J
326 -0.139886373 M
327 1.721754523 w
328 -0.676887487 q
329 0.022655467 y
330 2.063708971 k
331 0.527333879 X
332 2.033538617 W
333 -0.371122612 f
334 -1.165702215 F
335 1.273530467 Q
336 0.793559692 l
337 0.748991909 k
338 -0.633145751 G
339 -0.784559910 g
340 1.476111045 X
341 0.354139275 X
342 -0.674359454 K
343 -1.365197168 k
344 -0.507931845 a
345 0.225120017 h
346 0.129927247 Y
347 0.173364552 q
348 0.559597543 H
349 -0.490257888 c
350 -1.284180535 B
351 0.619761307 D
352 0.500147736 f
353 -0.369340328 i
354 0.803197104 j
355 1.397572924 b
356 -0.953040286 p
357 -0.816560532 s
358 -1.051015931 j
359 -0.181634598 s
360 -0.147096216 d
361 0.027221709 U
362 0.620745888 F
363 0.262687984 v
364 1.380483335 U
365 0.067567313 X
366 1.751570602 W
367 0.438825748 M
368 1.808340312 a
369 0.605527586 m
370 -0.483390631 X
371 0.211558655 I
372 -0.310306157 m
373 1.929228659 q
374 -1.617544923 P
375 0.832937253 C
376 -1.354096351 v
377 0.864418238 y
378 -0.726033869 b
379 -0.081856841 m
380 -1.334423081 Q
381 1.349231266 d
382 -0.222002196 Q
383 0.096457879 D
384 -0.002604509 T
385 -0.071723054 s
386 -0.165322615 j
387 0.354317703 y
388 -2.132777022 E
389 0.746357122 R
390 -0.573555847 j
391 -0.985549065 Y
392 1.048377586 H
393 0.460067177 Q
394 -0.622628970 U
395 0.219134248 N
396 0.910607981 z
397 -1.419319633 e
398 0.797456230 K
399 -0.915363689 T
400 0.781248226 x
401 -0.465741769 b
402 0.347827955 t
403 -0.484938541 i
404 -2.615634813 k
405 0.636529569 P
406 0.822472348 C
407 -0.743673255 p
408 -0.471838816 v
409 1.125488678 i
410 -0.584376931 F
411 -0.045750751 E
412 0.375471677 r
413 0.439682384 B
414 -0.499024458 l
415 -0.462578656 Q
416 0.227108042 W
417 0.214445204 D
418 0.677033616 r
419 0.594437079 F
420 0.766851411 d
421 -0.366306336 O
422 -1.624552019 u
423 0.156704168 N
424 -0.178724694 P
425 -1.219080521 q
426 0.813037877 P
427 -0.040579485 I
428 0.899585211 s
429 -0.469361921 K
430 1.165576753 B
431 0.655512633 X
432 1.269338020 U
433 0.061847941 m
434 0.043032287 W
435 -1.205534717 Y
436 -1.669729379 q
437 0.472410435 C
438 -0.680110898 q
439 0.099689207 q
440 -1.517089491 j
441 -0.473095178 Q
442 -1.341674480 j
443 -0.547533492 m
444 -1.075333395 B
445 -0.516450797 W
446 -0.089872719 P
447 -0.116574984 Q
448 0.999104890 h
449 0.283193034 d
450 0.209168887 e
451 -0.244943703 W
452 0.369033996 h
453 -0.137591890 X
454 0.402650717 g
455 -1.643721269 q
456 0.552094263 L
457 -0.170055723 k
458 -0.085523014 c
459 0.161168458 d
460 0.026135651 d
461 0.006249210 n
462 0.418695142 V
463 0.619619112 k
464 0.732148800 V
465 -0.612258867 a
466 -1.484476305 G
467 0.106063420 P
468 0.890715422 R
469 0.737146085 N
470 1.337442914 k
471 -0.011583379 J
472 0.473525830 u
473 0.934309141 B
474 0.848466438 n
475 0.191603208 k
476 0.074001182 W
477 -0.774560825 R
478 0.194674439 D
479 -0.042914952 W
480 0.061507791 k
481 -0.339130690 S
482 1.216940233 b
483 -0.801135515 b
484 0.248687120 A
485 -0.090106558 g
486 0.056776855 G
487 0.756716250 n
488 -1.189236326 s
489 0.852729173 h
490 0.790296790 A
491 0.166906623 m
492 -1.485032283 B
493 0.473288640 U
494 -0.337701505 x
495 0.400640444 G
496 -0.549024347 r
497 -0.707923360 E
498 -0.363968437 o
499 -0.227005034 z
500 -0.507420238 d
501 1.090735900 M
502 0.487110952 Q
503 1.598110625 l
504 -0.414871492 g
505 0.509773619 H
506 -1.547784889 H
507 -0.267903308 A
508 0.454836389 h
509 -1.294239517 i
510 1.427798866 P
511 0.626832625 x
512 -0.270171961 v
513 0.683703705 e
514 -0.242178967 e
515 1.113898200 I
516 -1.106326958 L
517 -1.829972604 t
518 -0.957891628 n
519 1.150266879 D
520 -1.020600052 G
521 0.418048137 S
522 -0.488215952 L
523 -1.239629812 r
524 0.512532540 m
525 -0.847790739 p
526 1.140368473 g
527 -1.147748779 w
528 -1.605929073 V
529 0.938544348 s
530 0.725422473 b
531 -0.649194418 b
532 1.508653353 W
533 -1.092310046 k
534 1.274452030 Y
535 -1.157686536 S
536 0.168140225 k
537 0.674805533 o
538 1.018426747 h
539 1.637443236 D
540 0.347698826 X
541 -0.232420640 H
542 -0.549612574 Y
543 -2.293790187 t
544 -0.248125746 h
545 -1.131133447 x
546 1.035848342 a
547 -0.915169225 F
548 1.472984395 C
549 -0.222447120 e
550 -1.853596798 E
551 1.366062746 R
552 -0.089091978 U
553 -1.374126277 h
554 0.803198350 Y
555 -2.331352954 J
556 1.095309539 n
557 0.667086890 U
558 0.246743847 v
559 -0.116318663 k
560 0.375461995 W
561 0.656705800 Q
562 0.795290592 Y
563 0.708626783 m
564 -1.271059359 s
565 -0.007916019 V
566 -0.150617692 y
567 0.711100695 h
568 0.408747297 p
569 -0.744456680 r
570 1.231652156 A
571 1.305571610 T
572 0.188627294 R
573 0.088461302 I
574 -0.468062191 j
575 -1.270152051 M
576 2.710786854 u
577 -0.212354187 s
578 0.856251542 S
579 0.954718887 f
580 -1.205852934 R
581 -2.389677976 i
582 -0.011964941 f
583 -1.951361060 k
584 -0.978222506 x
585 1.621375845 u
586 0.534849442 E
587 1.065981885 L
588 -1.864112996 r
589 -1.189043591 p
590 1.495348409 A
591 1.298833153 D
592 0.109554846 N
593 0.896960776 G
594 -0.384848805 F
595 -0.437187705 c
596 0.650794390 f
597 0.762920514 k
598 0.652238676 E
599 0.214962766 r
600 1.314182573 s
601 -0.989364217 Y
602 -0.456532942 N
603 2.698417145 g
604 0.664847464 A
605 1.318593908 i
606 0.523739859 r
607 -0.546917164 o
608 -0.689931290 B
609 1.279365610 C
610 -0.924985618 b
611 -0.387369247 k
612 -0.143435021 s
613 -1.440571799 F
614 0.555061899 a
615 -0.849209172 l
616 -0.645695105 a
617 -0.385508184 f
618 1.135516767 x
619 0.221361985 p
620 -0.676484388 a
621 -0.582881481 i
622 1.143699449 q
623 1.385825299 j
624 1.178294757 p
625 -1.644661538 K
626 1.292093937 t
627 -1.368298339 Y
628 -0.687719956 h
629 -0.352599811 l
630 -0.483822982 H
631 -1.012401042 n
632 -0.103346946 q
633 -0.298277633 x
634 -0.529234919 H
635 0.735607515 p
636 -0.953478972 q
637 0.049777009 s
638 1.794968256 K
639 -0.602398767 u
640 1.348821784 u
641 0.074252797 G
642 2.428886010 l
643 -0.858069520 A
644 0.996619446 d
645 0.132223805 K
646 -0.402397931 i
647 -0.404545474 z
648 -0.482684533 T
649 0.301099634 B
650 0.092504753 L
651 -1.041226447 j
652 0.275762503 R
653 -0.481135546 M
654 1.521959621 z
655 0.013970784 P
656 -1.164906574 K
657 -0.769764197 R
658 -0.407098360 U
659 0.692137313 i
660 -1.458167515 Q
661 -0.270806113 F
662 0.485273148 J
663 0.334282077 J
664 0.637046039 g
665 -0.706806028 i
666 -0.159734655 T
667 0.019349640 T
668 0.921840266 s
669 1.161080626 X
670 -0.841260681 m
671 0.863261918 c
672 -0.669842304 G
673 1.064663135 m
674 -0.541715453 F
675 0.385597180 T
676 0.832636697 x
677 -0.816978156 j
678 -0.439431160 F
679 1.182698479 P
680 -2.641130197 h
681 -3.425794969 a
682 -0.264003151 N
683 0.814116714 O
684 -0.518768072 s
685 -0.813826981 o
686 1.621309943 Q
687 0.046368720 Q
688 0.831397987 J
689 -0.359076716 L
690 0.176302230 t
691 -0.280269713 T
692 -0.875091222 I
693 -0.184881146 a
694 -0.187893100 r
695 -0.010546753 V
696 -0.811666823 T
697 -0.725640883 K
698 -0.895899393 v
699 0.302912213 T
700 -0.074020441 j
701 1.636457126 P
702 -0.788059980 q
703 -0.772041302 Q
704 -0.471964463 y
705 -0.714632594 l
706 0.253616276 X
707 1.548813503 E
708 -0.206721030 H
709 0.174694728 A
710 -0.408169545 k
711 1.822220987 d
712 -0.548483289 V
713 -0.958250202 x
714 0.863268800 B
715 -0.435127058 i
716 0.708791473 f
717 -0.274634384 p
718 0.641979800 k
719 -0.854322096 l
720 -0.844374342 P
721 -0.736718257 u
722 -0.085917149 L
723 1.016965933 z
724 0.210994190 g
725 -0.675013360 u
726 1.036072152 E
727 0.116366153 H
728 -1.259644944 C
729 -1.123268429 K
730 -0.921403895 M
731 -1.237759010 W
732 -0.249523600 c
733 0.835099880 S
734 1.767077902 p
735 0.542571429 o
736 0.037910229 R
737 1.247946937 Y
738 0.375565640 z
739 -0.287372340 D
740 1.354784408 I
741 0.654531640 p
742 0.852835790 X
743 -1.267447689 S
744 -0.412240791 f
745 1.784815075 j
746 0.643970200 L
747 -0.265997939 O
748 1.085963717 L
749 0.608321456 c
750 1.129974073 n
751 0.808468735 i
752 0.946144988 v
753 0.216468574 v
754 -0.193555197 Q
755 -0.905116741 S
756 -0.290976421 p
757 0.209083481 k
758 -1.262641657 G
759 -0.156325792 V
760 -0.073991477 c
761 -0.268585679 G
762 0.388661026 W
763 0.101443427 p
764 -0.718004416 X
765 0.454448868 a
766 0.211720060 S
767 -0.595319876 G
768 -0.785499707 Y
769 0.487139831 n
770 0.851797480 A
771 -0.702159642 y
772 0.646773584 w
773 -0.266816670 Z
774 1.918694607 k
775 -0.282033738 a
776 0.419856842 r
777 0.521949165 y
778 0.931677135 f
779 0.615630688 D
780 2.026088678 C
781 -0.632827039 d
782 0.282809000 Z
783 1.052449643 M
784 0.759557384 y
785 1.132104018 g
786 -1.118075823 A
787 -1.155696808 F
788 1.110308124 z
789 -0.332546703 E
790 -1.993530463 B
791 -0.323993855 Q
792 1.003071443 T
793 0.530573694 r
794 0.484398124 e
795 -2.946079952 d
796 -1.565539231 U
797 -1.066983422 Y
798 -0.015828641 T
799 -0.088380813 H
800 0.281055063 i
801 0.931979885 L
802 0.318145439 E
803 0.548225298 k
804 -1.206661828 L
805 0.375261135 q
806 0.246320102 n
807 1.077560684 L
808 -2.005814138 K
809 1.437339941 S
810 -0.999633906 S
811 1.354633971 a
812 -0.023363317 j
813 1.783377354 D
814 1.034842029 a
815 -0.407835639 U
816 0.669910625 o
817 -1.578196504 C
818 0.556915498 y
819 -1.520008319 B
820 -0.941989039 b
821 -0.028672079 X
822 1.136414136 q
823 1.373862918 x
824 -0.174444736 O
825 -0.951346908 T
826 -0.032294895 h
827 1.187572347 o
828 1.454372142 K
829 -1.073661686 T
830 -1.509977819 p
831 -0.808428189 r
832 -0.206626698 X
833 0.401802019 U
834 -0.149867197 N
835 0.603452317 r
836 -0.418550389 K
837 0.069911879 u
838 0.168385991 x
839 -1.033971713 d
840 0.666071218 H
841 -0.443262447 H
842 1.314743287 W
843 1.088574490 Y
844 0.090566315 b
845 -0.689411097 i
846 -0.560099886 f
847 -0.095166423 Y
848 1.659329989 b
849 2.115119517 b
850 -0.095349503 z
851 0.151420138 f
852 -0.599216235 Z
853 -1.044039888 X
854 -0.977129068 S
855 -0.836369219 t
856 -0.138905155 T
857 -0.320798364 x
858 -0.178019909 A
859 -0.385612679 i
860 -1.130974941 Z
861 -1.516166603 S
862 -1.188933655 A
863 -0.591099108 O
864 -0.704681440 C
865 -2.100540282 L
866 -1.083926348 p
867 0.924310269 b
868 -0.744559010 S
869 1.341053320 m
870 -0.066515980 t
871 -0.185075317 e
872 0.122222539 R
873 0.110523846 J
874 -0.624955176 O
875 -0.226141564 p
876 -0.150442113 T
877 -1.554956472 T
878 0.817585727 z
879 -1.093428024 C
880 0.083669041 a
881 -0.363984705 H
882 0.857606314 h
883 -0.578666637 P
884 0.695382031 M
885 0.223144558 a
886 -0.910722471 t
887 0.588875844 u
888 0.030301076 u
889 -0.887624024 F
890 -1.451915427 i
891 -0.325353926 g
892 0.574979467 T
893 0.577286339 X
894 0.942858001 s
895 0.687924774 V
896 0.268485424 O
897 -0.586009515 z
898 0.097277897 L
899 -2.161857030 V
900 -0.344354570 z
901 -1.987384490 l
902 0.651788848 p
903 -0.135703179 a
904 -0.350016712 S
905 1.019608983 B
906 -0.058168663 Q
907 -1.412936402 z
908 0.202399205 Y
909 0.238798161 A
910 1.707516900 S
911 1.018748162 I
912 -0.233390927 R
913 -0.054426120 c
914 -1.167133086 q
915 -0.961944317 m
916 0.145154890 W
917 -1.329591799 M
918 0.685257271 j
919 -2.165137353 n
920 1.537077395 D
921 3.170655395 o
922 -0.583085953 A
923 -0.200506685 D
924 -0.482447627 f
925 0.799978501 b
926 0.269259107 Y
927 0.905755656 t
928 0.210174750 h
929 0.267648982 E
930 -0.381440151 D
931 -0.288765707 r
932 0.464681588 O
933 -0.486026781 i
934 -1.680386937 w
935 -0.287732489 k
936 2.566538763 j
937 -1.182550359 R
938 0.517304612 S
939 2.661175755 Z
940 -0.836474378 k
941 1.498772143 v
942 0.331586499 E
943 -0.625229890 B
944 -0.855253562 e
945 1.359865865 t
946 0.414675823 H
947 -1.055197971 R
948 0.075003209 t
949 -0.100969184 f
950 1.106992628 B
951 0.827849241 J
952 -0.298887474 S
953 -0.843170414 D
954 -0.668100724 p
955 -1.271991797 O
956 0.262855822 A
957 -0.431714283 g
958 -0.120455116 U
959 -0.048794095 h
960 0.135342470 D
961 1.295291676 o
962 -0.752691282 o
963 0.071833685 h
964 -0.496364345 y
965 -0.184916864 b
966 1.717031332 q
967 -0.685297174 I
968 -0.739211749 Y
969 -1.204921890 X
970 0.511363389 h
971 -1.459805693 L
972 -0.990872745 B
973 2.458072252 Z
974 1.431963877 w
975 1.880584857 z
976 0.218424403 i
977 0.743657086 c
978 -0.517339639 o
979 0.194354308 l
980 0.779488329 w
981 -0.142717299 R
982 -0.776088791 h
983 -0.751613809 w
984 -0.984560457 W
985 -0.169688936 G
986 0.862051110 u
987 -0.459938069 l
988 0.487569343 t
989 0.500037106 q
990 0.373956329 N
991 -0.622625525 c
992 -0.892980978 r
993 -0.169332737 x
994 -1.336198951 j
995 -1.938961635 y
996 -1.994658725 S
997 0.460565354 r
998 0.701467161 y
999 1.222810195 K
1000 1.423615900 lQuick and easy. I think py_to_r is supposed to do some of this, but I can never get it to work. I think maybe it would make more sense in a script where we’re moving back and forth than here where we have separate code chunks?
Now, with time
Let’s add a column of dates to simpledf. First, create the dates, then add to simpledf.
```{python}
import datetime
dates = []
d1 = datetime.datetime.strptime('2000-01-01', '%Y-%m-%d')
# Because i starts at 0, the first loop is the start date
for i in range(1000):
# Add i days to the start date
day_new = d1 + datetime.timedelta(days=i)
# Append the current date string to the list of dates
dates.append(day_new)
timedf = simpledf.assign(date = dates)
# Try another way too
timedf['date2'] = pd.to_datetime(timedf['date'])
```Now, when we bring it into R, it just works? That’s not at all what happened to me when I had the original issue.
```{r}
timedfR <- py$timedf
# tibble::as_tibble(timedfR)
```For future reference
So, I can’t seem to replicate the issue. Previously, the datetime col came in as a list-col into a tibble, and was wrapped in a python environment. It was possible to parse it with purrr (or lapply, but purrr was much faster (and weirdly, furrr was slower). Not running, because this isn’t in a python env, and so this doesn’t actually work.
```{r}
#| eval: false
demodates <- timedfR$date
rdates <- purrr::map(demodates, py_to_r) %>%
tibble(.name_repair = ~'Date') %>%
unnest(cols = Date)
```But what was much faster was to convert the column to strings in python with
```{python}
timedf['date_str'] = timedf['date'].astype(str)
timedf.info()
```<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1000 entries, 0 to 999
Data columns (total 5 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 rand_nums 1000 non-null float64
1 rand_chars 1000 non-null object
2 date 1000 non-null datetime64[ns]
3 date2 1000 non-null datetime64[ns]
4 date_str 1000 non-null object
dtypes: datetime64[ns](2), float64(1), object(2)
memory usage: 39.2+ KBAnd then lubridate back to dates in R
```{r}
timedfRstr <- py$timedf
timedfRstr <- dplyr::select(timedfRstr, -date) |>
dplyr::mutate(date = lubridate::ymd(date_str))
str(timedfRstr)
```'data.frame': 1000 obs. of 5 variables:
$ rand_nums : num 0.133 -0.614 -0.62 -2.494 0.805 ...
$ rand_chars: chr "H" "w" "N" "N" ...
$ date2 : POSIXct, format: "2000-01-01 11:00:00" "2000-01-02 11:00:00" ...
$ date_str : chr "2000-01-01" "2000-01-02" "2000-01-03" "2000-01-04" ...
$ date : Date, format: "2000-01-01" "2000-01-02" ...
- attr(*, "pandas.index")=RangeIndex(start=0, stop=1000, step=1)