Are you over 18 and want to see adult content?
More Annotations
![A complete backup of voluntariresidence.ro](https://www.archivebay.com/archive5/images/76153de6-cb6d-43b1-bc18-250af18d0932.png)
A complete backup of voluntariresidence.ro
Are you over 18 and want to see adult content?
![A complete backup of seguroscatalanaoccidente.com](https://www.archivebay.com/archive5/images/574bde0e-90d8-4ad4-8956-bbede4610aab.png)
A complete backup of seguroscatalanaoccidente.com
Are you over 18 and want to see adult content?
![A complete backup of occupationalinfo.org](https://www.archivebay.com/archive5/images/2d10c062-bc41-4eb8-b576-23b3dfa5c2b4.png)
A complete backup of occupationalinfo.org
Are you over 18 and want to see adult content?
![A complete backup of heritageparts.com](https://www.archivebay.com/archive5/images/8d373a95-64a7-4141-a963-c51201a79e6f.png)
A complete backup of heritageparts.com
Are you over 18 and want to see adult content?
![A complete backup of appsinformative.com](https://www.archivebay.com/archive5/images/dd4061ed-e13e-48ed-a216-bf01b92c70ab.png)
A complete backup of appsinformative.com
Are you over 18 and want to see adult content?
![A complete backup of oceanweather.com](https://www.archivebay.com/archive5/images/deffb879-ad50-401b-b8aa-cc8d45e9e54b.png)
A complete backup of oceanweather.com
Are you over 18 and want to see adult content?
![A complete backup of commerzbank-arena.de](https://www.archivebay.com/archive5/images/853d8031-8653-4d02-b876-5c32aa0d78b4.png)
A complete backup of commerzbank-arena.de
Are you over 18 and want to see adult content?
![A complete backup of mobilesales.com.br](https://www.archivebay.com/archive5/images/48ae33c4-47f8-4cec-a84e-9513e2ead4e0.png)
A complete backup of mobilesales.com.br
Are you over 18 and want to see adult content?
![A complete backup of topleatherjackets.com](https://www.archivebay.com/archive5/images/8ba07c90-15d7-4ea9-84dd-77c696dc5907.png)
A complete backup of topleatherjackets.com
Are you over 18 and want to see adult content?
Favourite Annotations
![Geek Store – Váš Geek pre Hry, Filmy a Počítače](https://www.archivebay.com/archive/65b27ec5-8d97-490b-af27-e2fe1f20cde5.png)
Geek Store – Váš Geek pre Hry, Filmy a Počítače
Are you over 18 and want to see adult content?
![Vintage Digital Studio Effects from the 70s. 80s, 90s and Now!](https://www.archivebay.com/archive/ea6a79f0-5d80-46b5-beeb-e620baf8e9c2.png)
Vintage Digital Studio Effects from the 70s. 80s, 90s and Now!
Are you over 18 and want to see adult content?
![Главная Центр Пескоструйной и Антикоррозийной обработки](https://www.archivebay.com/archive/4e71a716-ef54-4c6d-aa9a-cf28ba2b4da1.png)
Главная Центр Пескоструйной и Антикоррозийной обработки
Are you over 18 and want to see adult content?
![Tutorial Blog, SEO, & Internet Marketing - IrvanTaufik.me](https://www.archivebay.com/archive/3d0296f1-b73f-40d8-b6d1-e09f7919f2b9.png)
Tutorial Blog, SEO, & Internet Marketing - IrvanTaufik.me
Are you over 18 and want to see adult content?
![The Mezzofanti Guild - Language Learning Made Simple](https://www.archivebay.com/archive/376263bd-6a57-4b53-9bb0-531562b758bf.png)
The Mezzofanti Guild - Language Learning Made Simple
Are you over 18 and want to see adult content?
![Price Spectre - Dynamic Pricing for eBay](https://www.archivebay.com/archive/a0abc1a2-90fa-470b-bda5-ed17b64b30e4.png)
Price Spectre - Dynamic Pricing for eBay
Are you over 18 and want to see adult content?
![Bündnis 90 / Die Grünen - Kreisverband Aachen](https://www.archivebay.com/archive/1d66dcab-f803-41d4-a9e4-3d831479cece.png)
Bündnis 90 / Die Grünen - Kreisverband Aachen
Are you over 18 and want to see adult content?
![Центр генеалогических исследований - поиск родственников и предков по фамилии](https://www.archivebay.com/archive/0509bf19-6944-450d-94fc-99154a323e66.png)
Центр генеалогических исследований - поиск родственников и предков по фамилии
Are you over 18 and want to see adult content?
Text
construction.
THE ULTRARICH'S DIRTY SECRET: NOT PAYING TAXES The tax situation in the United States is pretty messed up. Much ado has been made over the last week about AOC suggesting bringing back the pre-Reagan era 70% marginal tax rate on regular income over $10,000,000. This means that if you made $12 million in normal income SPYING ON INSTANCE METHODS WITH PYTHON'S MOCK MODULE Spying on instance methods with Python's mock module. Python's mock module ( unittest.mock in Python 3.3 and higher) allows you to observe parameters passed to functions. I'm a little slow, so I had to dig around to figure out how to do this. Let's say you have a class: Now, let's suppose you are testing the functionality of ProductionClass DO AVERAGE CONSUMERS STILL NEED DROPBOX? TL;DR: At the risk of stating the obvious, manual management of files on disks now in 2016 is increasingly old-fashioned and largely unnecessary, especially among the non-technorati. Encapsulated / managed cloud services and consumer web applications have made it anachronistic for most normal people. Whether this is a good thing can be debated, but it is happening nonetheless. WHY PANDAS USERS SHOULD BE EXCITED ABOUT APACHE ARROW Why pandas users should be excited about Apache Arrow. I'm super excited to be involved in the new open source Apache Arrow community initiative. For Python (and R, too!), it will help enable. Closer to native performance Python extensions for big data systems like Apache Spark. New in-memory analytics functionality for nested / JSON-likedata.
WES MCKINNEYABOUTTALKSBOOKBLOG Wes McKinney. I published the first edition in 2012, and the 2nd edition was published in 2017. Click to read more. PYTHON FOR DATA ANALYSIS BOOK Python for Data Analysis Book. The 2nd Edition of my book was released digitally on September 25, 2017, with print copies shipping a few weeks later. The 1st Edition was published in October, 2012. STREAMING COLUMNAR DATA WITH APACHE ARROW Over the past couple weeks, Nong Li and I added a streaming binary format to Apache Arrow, accompanying the existing random access / IPC file format. We have implementations in Java and C++, plus Python bindings. In this post, I explain how the format works and show how you can achieve very high data throughput to pandas DataFrames. NATIVE HADOOP FILE SYSTEM (HDFS) CONNECTIVITY IN PYTHON There have been many Python libraries developed for interacting with the Hadoop File System, HDFS, via its WebHDFS gateway as well as its native Protocol Buffers-based RPC interface. I'll give you an overview of what's out there and show some engineering I've been doing to offer a high performance HDFS interface within the developing Arrow ecosystem. This blog is a follow up to my APACHE ARROW AND THE "10 THINGS I HATE ABOUT PANDAS" FROM ARROW TO PANDAS AT 10 GIGABYTES PER SECOND To go back to pandas-land, call the table's to_pandas method. This supports a multithreaded conversion, so let's do a single-threaded conversion for comparison: >>> %timeit df2 = table.to_pandas(nthreads=1) 10 loops, best of 3: 158 ms per loop. This is 6.33 GB/s, or about 20% slower than purely memcpy-basedconstruction.
THE ULTRARICH'S DIRTY SECRET: NOT PAYING TAXES The tax situation in the United States is pretty messed up. Much ado has been made over the last week about AOC suggesting bringing back the pre-Reagan era 70% marginal tax rate on regular income over $10,000,000. This means that if you made $12 million in normal income SPYING ON INSTANCE METHODS WITH PYTHON'S MOCK MODULE Spying on instance methods with Python's mock module. Python's mock module ( unittest.mock in Python 3.3 and higher) allows you to observe parameters passed to functions. I'm a little slow, so I had to dig around to figure out how to do this. Let's say you have a class: Now, let's suppose you are testing the functionality of ProductionClass DO AVERAGE CONSUMERS STILL NEED DROPBOX? TL;DR: At the risk of stating the obvious, manual management of files on disks now in 2016 is increasingly old-fashioned and largely unnecessary, especially among the non-technorati. Encapsulated / managed cloud services and consumer web applications have made it anachronistic for most normal people. Whether this is a good thing can be debated, but it is happening nonetheless. WHY PANDAS USERS SHOULD BE EXCITED ABOUT APACHE ARROW Why pandas users should be excited about Apache Arrow. I'm super excited to be involved in the new open source Apache Arrow community initiative. For Python (and R, too!), it will help enable. Closer to native performance Python extensions for big data systems like Apache Spark. New in-memory analytics functionality for nested / JSON-likedata.
ABOUT - WES MCKINNEY Biography: Wes McKinney is an open source software developer focusing on data analysis tools. He created the Python pandas project and is a co-creator of Apache Arrow, his current development focus. He authored 2 editions of the reference book "Python for Data Analysis". Wes is a Member of The Apache Software Foundation and also a PMC memberPRESENTATIONS
Upcoming 2020-02-26: ScaledML (Mountain View, CA) 2020-05-08: NYC R Conference (New York, NY) 2020-05-26: Ray Summit (San Francisco, CA) Past 2020-02-08: PyCon Colombia (Medellín, CO) Slides 2019-10-22: OmniSci Converge (Mountain View, CA) 2019-05-10: NYC R Conference (New York, New York) Apache Arrow: Leveling Up the Data Science StackSlides 2019-01-11
ARCHIVES - WES MCKINNEY Fri 22 March 2013 I'm moving to San Francisco. And hiring. Mon 11 February 2013 Whirlwind tour of pandas in 10 minutes. Mon 10 December 2012 Update on upcoming pandas v0.10, new file parser, other performance wins. Thu 04 October 2012 A new high performance, memory-efficient file parser engine for pandas. APACHE ARROW AND THE "10 THINGS I HATE ABOUT PANDAS" This post is the first of many to come on Apache Arrow, pandas, pandas2, and the general trajectory of my work in recent times and into the foreseeable future. This is a bit of a read and overall fairly technical, but if interested I encourage you to take the time THE ULTRARICH'S DIRTY SECRET: NOT PAYING TAXES The tax situation in the United States is pretty messed up. Much ado has been made over the last week about AOC suggesting bringing back the pre-Reagan era 70% marginal tax rate on regular income over $10,000,000. This means that if you made $12 million in normal income in a single year, you would pay 70% of the last $2 million to thefederal government.
AVOID UNSIGNED INTEGERS IN C++ IF YOU CAN Unsigned integers (size_t, uint32_t, and friends) can be hazardous, as signed-to-unsigned integer conversions can happen without so much as a compiler warning.An example: size_t as an index variable Occasionally, discussions come up about using unsigned integers as index variables for STL containers (whose size() attribute is unsigned). So the debate is between effectively these two EVEN EASIER FREQUENCY TABLES IN PANDAS 0.7.0 Even easier frequency tables in pandas 0.7.0. I put in a little work on a new crosstab function in the main pandas namespace. It's basically a convenient shortcut to calling pivot_table to make it easy to compute cross-tabulations for a set of factors using pandas DataFrame or even vanilla NumPy arrays! Here's an example: LEAVING NYC FOR NASHVILLE For ten out of the last eleven years, I've lived in two places: New York City and San Francisco. The last two years have been in NYC. After founding Ursa Labs, a not-for-profit open source development group, I felt it was time to make my home somewhere that isn't either of those places.After some contemplation and consulting many friends, I decided on Nashville, Tennessee. SPEEDING UP PANDAS'S FILE PARSERS WITH CYTHON For multi-dimensional arrays, you specify the number of dimensions in the buffer. and pass multiple indexes (Py_ssize_t is the proper C "index type" to use).I'll demonstrate this in: Converting rows to columns faster than zip(*rows) FEATHER FORMAT UPDATE: WHENCE AND WHITHER? Earlier this year, development for the Feather file format moved to the Apache Arrow codebase. I will explain how this has already affected Feather and what to WES MCKINNEYABOUTTALKSBOOKBLOG Wes McKinney. I published the first edition in 2012, and the 2nd edition was published in 2017. Click to read more. PYTHON FOR DATA ANALYSIS BOOK Python for Data Analysis Book. The 2nd Edition of my book was released digitally on September 25, 2017, with print copies shipping a few weeks later. The 1st Edition was published in October, 2012. NATIVE HADOOP FILE SYSTEM (HDFS) CONNECTIVITY IN PYTHON There have been many Python libraries developed for interacting with the Hadoop File System, HDFS, via its WebHDFS gateway as well as its native Protocol Buffers-based RPC interface. I'll give you an overview of what's out there and show some engineering I've been doing to offer a high performance HDFS interface within the developing Arrow ecosystem. This blog is a follow up to my STREAMING COLUMNAR DATA WITH APACHE ARROW Over the past couple weeks, Nong Li and I added a streaming binary format to Apache Arrow, accompanying the existing random access / IPC file format. We have implementations in Java and C++, plus Python bindings. In this post, I explain how the format works and show how you can achieve very high data throughput to pandas DataFrames. FROM ARROW TO PANDAS AT 10 GIGABYTES PER SECOND To go back to pandas-land, call the table's to_pandas method. This supports a multithreaded conversion, so let's do a single-threaded conversion for comparison: >>> %timeit df2 = table.to_pandas(nthreads=1) 10 loops, best of 3: 158 ms per loop. This is 6.33 GB/s, or about 20% slower than purely memcpy-basedconstruction.
APACHE ARROW AND THE "10 THINGS I HATE ABOUT PANDAS" DO AVERAGE CONSUMERS STILL NEED DROPBOX? TL;DR: At the risk of stating the obvious, manual management of files on disks now in 2016 is increasingly old-fashioned and largely unnecessary, especially among the non-technorati. Encapsulated / managed cloud services and consumer web applications have made it anachronistic for most normal people. Whether this is a good thing can be debated, but it is happening nonetheless. AVOID UNSIGNED INTEGERS IN C++ IF YOU CAN Unsigned integers (size_t, uint32_t, and friends) can be hazardous, as signed-to-unsigned integer conversions can happen without so much as a compiler warning.An example: size_t as an index variable Occasionally, discussions come up about using unsigned integers as index variables for STL containers (whose size() attribute is unsigned). So the debate is between effectively these two SPYING ON INSTANCE METHODS WITH PYTHON'S MOCK MODULE Spying on instance methods with Python's mock module. Python's mock module ( unittest.mock in Python 3.3 and higher) allows you to observe parameters passed to functions. I'm a little slow, so I had to dig around to figure out how to do this. Let's say you have a class: Now, let's suppose you are testing the functionality of ProductionClass WHY PANDAS USERS SHOULD BE EXCITED ABOUT APACHE ARROW Why pandas users should be excited about Apache Arrow. I'm super excited to be involved in the new open source Apache Arrow community initiative. For Python (and R, too!), it will help enable. Closer to native performance Python extensions for big data systems like Apache Spark. New in-memory analytics functionality for nested / JSON-likedata.
WES MCKINNEYABOUTTALKSBOOKBLOG Wes McKinney. I published the first edition in 2012, and the 2nd edition was published in 2017. Click to read more. PYTHON FOR DATA ANALYSIS BOOK Python for Data Analysis Book. The 2nd Edition of my book was released digitally on September 25, 2017, with print copies shipping a few weeks later. The 1st Edition was published in October, 2012. NATIVE HADOOP FILE SYSTEM (HDFS) CONNECTIVITY IN PYTHON There have been many Python libraries developed for interacting with the Hadoop File System, HDFS, via its WebHDFS gateway as well as its native Protocol Buffers-based RPC interface. I'll give you an overview of what's out there and show some engineering I've been doing to offer a high performance HDFS interface within the developing Arrow ecosystem. This blog is a follow up to my STREAMING COLUMNAR DATA WITH APACHE ARROW Over the past couple weeks, Nong Li and I added a streaming binary format to Apache Arrow, accompanying the existing random access / IPC file format. We have implementations in Java and C++, plus Python bindings. In this post, I explain how the format works and show how you can achieve very high data throughput to pandas DataFrames. FROM ARROW TO PANDAS AT 10 GIGABYTES PER SECOND To go back to pandas-land, call the table's to_pandas method. This supports a multithreaded conversion, so let's do a single-threaded conversion for comparison: >>> %timeit df2 = table.to_pandas(nthreads=1) 10 loops, best of 3: 158 ms per loop. This is 6.33 GB/s, or about 20% slower than purely memcpy-basedconstruction.
APACHE ARROW AND THE "10 THINGS I HATE ABOUT PANDAS" DO AVERAGE CONSUMERS STILL NEED DROPBOX? TL;DR: At the risk of stating the obvious, manual management of files on disks now in 2016 is increasingly old-fashioned and largely unnecessary, especially among the non-technorati. Encapsulated / managed cloud services and consumer web applications have made it anachronistic for most normal people. Whether this is a good thing can be debated, but it is happening nonetheless. AVOID UNSIGNED INTEGERS IN C++ IF YOU CAN Unsigned integers (size_t, uint32_t, and friends) can be hazardous, as signed-to-unsigned integer conversions can happen without so much as a compiler warning.An example: size_t as an index variable Occasionally, discussions come up about using unsigned integers as index variables for STL containers (whose size() attribute is unsigned). So the debate is between effectively these two SPYING ON INSTANCE METHODS WITH PYTHON'S MOCK MODULE Spying on instance methods with Python's mock module. Python's mock module ( unittest.mock in Python 3.3 and higher) allows you to observe parameters passed to functions. I'm a little slow, so I had to dig around to figure out how to do this. Let's say you have a class: Now, let's suppose you are testing the functionality of ProductionClass WHY PANDAS USERS SHOULD BE EXCITED ABOUT APACHE ARROW Why pandas users should be excited about Apache Arrow. I'm super excited to be involved in the new open source Apache Arrow community initiative. For Python (and R, too!), it will help enable. Closer to native performance Python extensions for big data systems like Apache Spark. New in-memory analytics functionality for nested / JSON-likedata.
ABOUT - WES MCKINNEY Biography: Wes McKinney is an open source software developer focusing on data analysis tools. He created the Python pandas project and is a co-creator of Apache Arrow, his current development focus. He authored 2 editions of the reference book "Python for Data Analysis". Wes is a Member of The Apache Software Foundation and also a PMC memberPRESENTATIONS
Upcoming 2020-02-26: ScaledML (Mountain View, CA) 2020-05-08: NYC R Conference (New York, NY) 2020-05-26: Ray Summit (San Francisco, CA) Past 2020-02-08: PyCon Colombia (Medellín, CO) Slides 2019-10-22: OmniSci Converge (Mountain View, CA) 2019-05-10: NYC R Conference (New York, New York) Apache Arrow: Leveling Up the Data Science StackSlides 2019-01-11
ARCHIVES - WES MCKINNEY Fri 22 March 2013 I'm moving to San Francisco. And hiring. Mon 11 February 2013 Whirlwind tour of pandas in 10 minutes. Mon 10 December 2012 Update on upcoming pandas v0.10, new file parser, other performance wins. Thu 04 October 2012 A new high performance, memory-efficient file parser engine for pandas. APACHE ARROW AND THE "10 THINGS I HATE ABOUT PANDAS" This post is the first of many to come on Apache Arrow, pandas, pandas2, and the general trajectory of my work in recent times and into the foreseeable future. This is a bit of a read and overall fairly technical, but if interested I encourage you to take the time AVOID UNSIGNED INTEGERS IN C++ IF YOU CAN Unsigned integers (size_t, uint32_t, and friends) can be hazardous, as signed-to-unsigned integer conversions can happen without so much as a compiler warning.An example: size_t as an index variable Occasionally, discussions come up about using unsigned integers as index variables for STL containers (whose size() attribute is unsigned). So the debate is between effectively these two EVEN EASIER FREQUENCY TABLES IN PANDAS 0.7.0 Even easier frequency tables in pandas 0.7.0. I put in a little work on a new crosstab function in the main pandas namespace. It's basically a convenient shortcut to calling pivot_table to make it easy to compute cross-tabulations for a set of factors using pandas DataFrame or even vanilla NumPy arrays! Here's an example: LEAVING NYC FOR NASHVILLE For ten out of the last eleven years, I've lived in two places: New York City and San Francisco. The last two years have been in NYC. After founding Ursa Labs, a not-for-profit open source development group, I felt it was time to make my home somewhere that isn't either of those places.After some contemplation and consulting many friends, I decided on Nashville, Tennessee. THE ULTRARICH'S DIRTY SECRET: NOT PAYING TAXES The tax situation in the United States is pretty messed up. Much ado has been made over the last week about AOC suggesting bringing back the pre-Reagan era 70% marginal tax rate on regular income over $10,000,000. This means that if you made $12 million in normal income in a single year, you would pay 70% of the last $2 million to thefederal government.
FILTERING OUT DUPLICATE PANDAS.DATAFRAME ROWS Sean Taylor recently alerted me to the fact that there wasn't an easy way to filter out duplicate rows in a pandas DataFrame. R has the duplicated function which serves this purpose quite nicely. The R method's implementation is kind of kludgy in my opinion (from "The data frame method works by pasting together a character representation of the rows"), but in any case I set about writing a FEATHER FORMAT UPDATE: WHENCE AND WHITHER? Earlier this year, development for the Feather file format moved to the Apache Arrow codebase. I will explain how this has already affected Feather and what to WES MCKINNEYABOUTTALKSBOOKBLOG I published the first edition in 2012, and the 2nd edition was published in 2017. Click to read more PYTHON FOR DATA ANALYSIS BOOK The 2nd Edition of my book was released digitally on September 25, 2017, with print copies shipping a few weeks later. The 1st Edition was published in October, 2012. Where to buy? Buy DRM-free PDF on eBooks.com Buy the 2nd Edition in English on Amazon.com in print or FROM ARROW TO PANDAS AT 10 GIGABYTES PER SECONDAPACHE ARROW EXAMPLESAPACHE ARROW JAVASCRIPTAPACHE ARROW SPARKAPACHE ARROW TUTORIAL At 9.71 GB/s, this is not far from saturating the main memory bandwidth on my consumer desktop hardware (but I am not an expert on this).. The performance benefits of multithreading can be more dramatic on other hardware. While the performance ratio on my desktop is only 1.53, on my (also quad-core) laptop it is 3.29.. Note that numeric data is a best-case scenario; string or binary data NATIVE HADOOP FILE SYSTEM (HDFS) CONNECTIVITY IN PYTHON There have been many Python libraries developed for interacting with the Hadoop File System, HDFS, via its WebHDFS gateway as well as its native Protocol Buffers-based RPC interface. I'll give you an overview of what's out there and show some engineering I've been doing to offer a high performance HDFS interface within the developing Arrow ecosystem. This blog is a follow up to my APACHE ARROW AND THE "10 THINGS I HATE ABOUT PANDAS" STREAMING COLUMNAR DATA WITH APACHE ARROW Over the past couple weeks, Nong Li and I added a streaming binary format to Apache Arrow, accompanying the existing random access / IPC file format. We have implementations in Java and C++, plus Python bindings. In this post, I explain how the format works and show how you can achieve very high data throughput to pandas DataFrames. DO AVERAGE CONSUMERS STILL NEED DROPBOX? TL;DR: At the risk of stating the obvious, manual management of files on disks now in 2016 is increasingly old-fashioned and largely unnecessary, especially among the non-technorati. Encapsulated / managed cloud services and consumer web applications have made it anachronistic for most normal people. Whether this is a good thing can be debated, but it is happening nonetheless. SPYING ON INSTANCE METHODS WITH PYTHON'S MOCK MODULE Now, let's suppose you are testing the functionality of ProductionClass, but you want to observe the parameters passed to your internal methods but still invoke those internal methods.I didn't find a lot of examples of this from my Google searches, so here is the solution using unittest.mock (or mock from PyPI if you're on LegacyPython 2.x):
AVOID UNSIGNED INTEGERS IN C++ IF YOU CAN Unsigned integers (size_t, uint32_t, and friends) can be hazardous, as signed-to-unsigned integer conversions can happen without so much as a compiler warning.An example: size_t as an index variable Occasionally, discussions come up about using unsigned integers as index variables for STL containers (whose size() attribute is unsigned). So the debate is between effectively these two WHY PANDAS USERS SHOULD BE EXCITED ABOUT APACHE ARROW I'm super excited to be involved in the new open source Apache Arrow community initiative. For Python (and R, too!), it will help enable Substantially improved data access speeds Closer to native performance Python extensions for big data systems like Apache Spark New in-memory analytics functionality for nested / JSON-like data There's plenty of places you can learn more about Arrow, but WES MCKINNEYABOUTTALKSBOOKBLOG Wes McKinney. I published the first edition in 2012, and the 2nd edition was published in 2017. Click to read more. PYTHON FOR DATA ANALYSIS BOOK Python for Data Analysis Book. The 2nd Edition of my book was released digitally on September 25, 2017, with print copies shipping a few weeks later. The 1st Edition was published in October, 2012. NATIVE HADOOP FILE SYSTEM (HDFS) CONNECTIVITY IN PYTHON There have been many Python libraries developed for interacting with the Hadoop File System, HDFS, via its WebHDFS gateway as well as its native Protocol Buffers-based RPC interface. I'll give you an overview of what's out there and show some engineering I've been doing to offer a high performance HDFS interface within the developing Arrow ecosystem. This blog is a follow up to my APACHE ARROW AND THE "10 THINGS I HATE ABOUT PANDAS" FROM ARROW TO PANDAS AT 10 GIGABYTES PER SECONDAPACHE ARROW EXAMPLESAPACHE ARROW JAVASCRIPTAPACHE ARROW SPARKAPACHE ARROW TUTORIAL To go back to pandas-land, call the table's to_pandas method. This supports a multithreaded conversion, so let's do a single-threaded conversion for comparison: >>> %timeit df2 = table.to_pandas(nthreads=1) 10 loops, best of 3: 158 ms per loop. This is 6.33 GB/s, or about 20% slower than purely memcpy-basedconstruction.
STREAMING COLUMNAR DATA WITH APACHE ARROW Over the past couple weeks, Nong Li and I added a streaming binary format to Apache Arrow, accompanying the existing random access / IPC file format. We have implementations in Java and C++, plus Python bindings. In this post, I explain how the format works and show how you can achieve very high data throughput to pandas DataFrames. DO AVERAGE CONSUMERS STILL NEED DROPBOX? TL;DR: At the risk of stating the obvious, manual management of files on disks now in 2016 is increasingly old-fashioned and largely unnecessary, especially among the non-technorati. Encapsulated / managed cloud services and consumer web applications have made it anachronistic for most normal people. Whether this is a good thing can be debated, but it is happening nonetheless. AVOID UNSIGNED INTEGERS IN C++ IF YOU CAN Unsigned integers (size_t, uint32_t, and friends) can be hazardous, as signed-to-unsigned integer conversions can happen without so much as a compiler warning.An example: size_t as an index variable Occasionally, discussions come up about using unsigned integers as index variables for STL containers (whose size() attribute is unsigned). So the debate is between effectively these two WHY PANDAS USERS SHOULD BE EXCITED ABOUT APACHE ARROW Why pandas users should be excited about Apache Arrow. I'm super excited to be involved in the new open source Apache Arrow community initiative. For Python (and R, too!), it will help enable. Closer to native performance Python extensions for big data systems like Apache Spark. New in-memory analytics functionality for nested / JSON-likedata.
SPYING ON INSTANCE METHODS WITH PYTHON'S MOCK MODULE Spying on instance methods with Python's mock module. Python's mock module ( unittest.mock in Python 3.3 and higher) allows you to observe parameters passed to functions. I'm a little slow, so I had to dig around to figure out how to do this. Let's say you have a class: Now, let's suppose you are testing the functionality of ProductionClass ABOUT - WES MCKINNEY statsmodels (website, code). I worked on time series models (e.g. VAR) and pandas integration. Long form biography. I'm an American computer programmer and the Director of Ursa Labs.I studied theoretical mathematics at MIT (graduating in late 2006) before becoming very interested in programming and tools for data analysis, especially for industry use cases, in 2007.PRESENTATIONS
Upcoming 2020-02-26: ScaledML (Mountain View, CA) 2020-05-08: NYC R Conference (New York, NY) 2020-05-26: Ray Summit (San Francisco, CA) Past 2020-02-08: PyCon Colombia (Medellín, CO) Slides 2019-10-22: OmniSci Converge (Mountain View, CA) 2019-05-10: NYC R Conference (New York, New York) Apache Arrow: Leveling Up the Data Science StackSlides 2019-01-11
EVEN EASIER FREQUENCY TABLES IN PANDAS 0.7.0 This makes it very easy to produce an easy-on-the-eyes frequency table. crosstab can also take NumPy arrays. Suppose we had 1 million draws from a normal distribution, and we wish to produce a histogram-like table showing the number of draws whose absolute values fall into the bins defined by .Also, let's divide thingsup by sign.
THE ULTRARICH'S DIRTY SECRET: NOT PAYING TAXES The tax situation in the United States is pretty messed up. Much ado has been made over the last week about AOC suggesting bringing back the pre-Reagan era 70% marginal tax rate on regular income over $10,000,000. This means that if you made $12 million in normal income in a single year, you would pay 70% of the last $2 million to thefederal government.
LEAVING NYC FOR NASHVILLE For ten out of the last eleven years, I've lived in two places: New York City and San Francisco. The last two years have been in NYC. After founding Ursa Labs, a not-for-profit open source development group, I felt it was time to make my home somewhere that isn't either of those places.After some contemplation and consulting many friends, I decided on Nashville, Tennessee. SPEEDING UP PANDAS'S FILE PARSERS WITH CYTHON For multi-dimensional arrays, you specify the number of dimensions in the buffer. and pass multiple indexes (Py_ssize_t is the proper C "index type" to use).I'll demonstrate this in: Converting rows to columns faster than zip(*rows) UPDATE ON UPCOMING PANDAS V0.10, NEW FILE PARSER, OTHER If you know much about this data set, you know most of these columns are not interesting to analyze. New in pandas v0.10 you can specify a subset of columns right in read_csv which results in both much faster parsing time and lower memory usage (since we're throwing away the data from the other columns after tokenizing the file): A ROADMAP FOR RICH SCIENTIFIC DATA STRUCTURES IN PYTHON Discussion thread on Hacker News So, this post is a bit of a brain dump on rich data structures in Python and what needs to happen in the very near future. I care about them for statistical computing (I want to build a statistical computing environment that trounces R) and DON'T SELL ON AMAZON Selling your stuff on Amazon is a losing game, and I don't recommend that you do it. Let me tell you what happened to me recently. I wasn't using my iPad so I decided to sell it on Amazon. COMPILING DATAFRAME CODE IS HARDER THAN IT LOOKS Many people have asked me about the proliferation of DataFrame APIs like Spark DataFrames, Ibis, Blaze, and others. As it turns out, executing pandas-like code in a scalable environment is a difficult compiler engineering problem to enable composable, imperative Python or R code to be translated into a SQL or Spark/MapReduce-likerepresentation.
Toggle navigation Wes McKinney* __About
* __Talks
* __Book
* __GitHub
* __Twitter
* __Blog
I published the first edition in 2012, and the 2nd edition was published in 2017. CLICK TO READ MOREBLOG
I write about technical topics (mostly related to Python programming), with occasional diversions into other things that interest me.ABOUT ME
Learn more about my present past life and projects I've worked on.PRESENTATIONS
I give frequent presentations and tutorials about my work at Python and industry conferences and user meetups.CODE
Most of my open source software is hosted on GitHub. -------------------------2019 Wes McKinney
__ Back to top
Details
Copyright © 2024 ArchiveBay.com. All rights reserved. Terms of Use | Privacy Policy | DMCA | 2021 | Feedback | Advertising | RSS 2.0