megaserg blog

How to speed up your PyTorch training

2020-10-13T00:00:00+00:00

If you’re training PyTorch models, you want to train as fast as possible. This means you can try more things faster and get better results. Let’s learn how to speedup your training!

This performance optimization guide is written with cloud-based, multi-machine multi-GPU training in mind, but can be useful even if all you have is a desktop with a single GPU under your bed.

Hardware factors

The training speed is fundamentally limited by three factors: CPU, GPU, and network.

CPU speed. Each training process uses CPUs 1) to receive, parse, preprocess, and batch data examples; 2) to execute some NN operations not supported by GPU; and 3) to postprocess metrics, save checkpoints, and write summaries. In cloud, you can choose how many CPU cores your machine has.
GPU speed. With DistributedDataParallel (DDP, current SotA for distributed training), we run one training PyTorch process per one GPU. Each training process uses GPU for most of the forward pass, backward pass, and weight update. In synchronized distributed training, GPUs communicate on each step to share gradients. In cloud, you can choose how many GPUs your machine has.
Network speed. Each training process uses network for data downloading and GPU communication. The “main” process also uploads model checkpoints and summaries. If you’re in Google Cloud and store your dataset on GCS, for example, each machine has about 10-100 Gbit/s ingress network which means it can download 1-10 GB/s from GCS. For comparison, my desktop can download from GCS at 100 MB/s. In cloud, you can scale total network bandwidth by adding more machines.

Note that each training process has its whole GPU to itself, but network and CPUs are shared equally between all training processes. So it’s useful to think about CPU/GPU ratio or network/GPU ratio.

The typical training step

Data loading (uses CPU + network)
- Download example (bound by network speed, parallelizable per example)
- Parse, preprocess, augment example (bound by CPU, parallelizable per example)
- Build a batch (bound by CPU, parallelizable per batch)
Memcpy batch to GPU (bound by PCI-e, parallelizable per GPU)
Training (uses GPU + network, parallelizable per batch)
- Forward pass, backward pass (bound by GPU)
- Share gradients (bound by network)
- Update weights (bound by GPU)
Postprocessing (uses CPU)
Save checkpoint and summaries (uses CPU)

Realizing the problem and ways to solve it

How do you even know the training or dataloading is slow? You can measure the time spent by the training process getting a batch from the dataloader and compare it to the total time of the step. Ideally, all dataloading is offloaded to background processes, so getting the next batch is nearly instantaneous and only take a few percents of total time. Another way is to print, e.g. every 10 batches, how long did it take to process them. That way you can see if you have suspicious pauses, e.g. when a new epoch starts.

We have several ways of making training faster:

we can do things in parallel
we can make individual things faster
we can avoid doing some things.

Parallelization

Ideally, CPU, GPU, and network should be used by the training simultaneously, which means each of them should always have something to do. Also, a machine has multiple CPUs, multiple GPUs, and multiple network connections. We want to get to the state where:

either all CPUs are 100% used
or all GPUs are 100% used (this is the ultimate goal as in the cloud, GPUs is the most expensive resource of all)
or the network is 100% used

If that’s not the case, e.g. your monitoring charts show CPU < 100%, GPU < 100%, and network < 1 GB/s, it means there is not enough parallelization, i.e. there is some “artificial” software limit or lock preventing full utilization of hardware. For the cloud, this also means we can’t scale vertically (by adding more CPUs/GPUs) but only horizontally (by adding more machines), and in fact, might need to downscale vertically if we can’t fully use the resources anyway.

How to see utilization on your desktop

CPU: I like htop (apt-get install -y htop). It shows how much each core is utilized with horizontal bars. Press Shift + H to hide individual threads. Press F5 to sort processes by CPU usage.
GPU: I like sudo nvtop (install). It shows graphical history of GPU and memory utilization, as well as processes using GPU. Alternatively, watch -n 1 nvidia-smi is okay.
Network: I like sudo iftop (apt-get install -y iftop). It shows the incoming/outgoing throughput per connection. The three columns are average bandwidth during the last 2, 10 and 40 seconds.
To isolate dataloading from the other things, I like to use a separate “dataloading benchmark” script which just fetches batches as fast as possible without engaging GPU. This is to reduce the number of moving parts.

Notes on parallelization: data loading

The main obstacle to dataloading parallelization is Python’s GIL: only one stream of computation can progress within a Python process at any moment. The way around it is to have multiple Python processes, or fall to C++ code which is not bound by GIL.
That’s why each training process has its own GPU to feed with data. We shard the dataset into as many pieces as we have GPUs. The processes don’t communicate except for sharing gradients, so GPUs can be kept busy in parallel.
- Note: you can tune this by having more or fewer GPUs. Remember to use torch.distributed.launch to enable multi-GPU training, otherwise only one GPU will be used.
With PyTorch DataLoader’s num_workers > 0, each training process offloads the dataloading to subprocesses. Each worker subprocess receives one batch worth of example indices, downloads them, preprocesses, and stacks the resulting tensors, and shares the resulting batch with the training process (by pickling into /dev/shm). The training process then only has to feed the GPU and postprocess. As a result, GPU computation and CPU computation overlap.
- Worker subprocesses are also working in parallel, making use of CPU and network. At any moment a process uses either only CPU or only network, so it’s even okay to have more processes than you have CPU cores.
- You can scale this parallelization by increasing num_workers, but too many processes exhaust /dev/shm space which manifests as “bus error” and training crashes. Remember that if the machine has several GPUs, each training process will launch num_workers processes!
- Worker processes are destroyed and recreated every epoch, although there is a way around that by wrapping the Sampler around.
- The workers need to assemble the whole batch, and they sequentially download and process the examples in the batch (sad!). Therefore, in the beginning of the epoch, when all worker processes start dataloading, you might observe a long pause while each of them is assembling their first batch. This pause goes away after the first batch, as long as the dataloading is not slower than training.
Network requests to GCS are issued in parallel but parallelism is bounded by the underlying connection pool size (by default 10). Increasing that did not achieve any speedup; likely because the library talking to GCS is in Python and therefore subject to GIL, and therefore unable to make use of more than 10 simultaneous connections.

Side note: Parquet format

You might consider to use Parquet as your dataset format. Parquet is a columnar format, meaning the records are written on disk not id1 | name1 | age1 | id2 | name2 | age2, but rather id1 | id2 | name1 | name2 | age1 | age2 (however, this holds true only within a “rowgroup” which is a group of several consecutive rows). This means columns of a record can be downloaded and parsed in parallel (unlike e.g. TFRecord format which is just one protobuf blob). You can read Parquet files with PyArrow library, which uses C++-based thread pool to read the columns in parallel. You can tune OMP_NUM_THREADS environment variable to increase the size of that thread pool, but too many threads can cause thrashing. Probably stick to default unless you know what you’re doing. Then, wrap the resulting dataset in DataLoader.
An alternative reader of Parquet is Petastorm, which in my opinion has made less optimal architectural choices compared to DataLoader. Petastorm also has a pool of workers which receive tasks (rowgroup ids), download rowgroups, and perform filtering and transforms (but not batching). There are two implementations of worker pool in Petastorm: thread pool and process pool. You can tune this with reader_pool_type= "thread" or "process".
- Thread pool is subject to GIL so there’s no CPU parallelization, and all computation happens in the same process that feeds the GPU, but I think network requests are parallelized (Python yields GIL when performing I/O).
- Process pool parallelizes both CPU and network usage. The worker processes send records back to the main process via ZeroMQ. This means records must be serialized, and the main process needs to spend time deserializing. The fastest way to do that is PyArrow serialization, but it can only serialize certain data types. The alternative is pickle serialization which understands all data types but is slower.
- Also, with Petastorm, the main training process is assembling the batches, which takes some CPU time and can leave GPU underutilized.

Notes on parallelization: memcpy

There is a region in RAM called “pinned memory” which is the waiting area for tensors before they can be placed on GPU. For faster CPU-to-GPU transfer, we can copy tensors in the pinned memory region in the background thread, before GPU asks for the next batch. This is available with pin_memory=True argument to PyTorch DataLoader.
- Documentation on pinned memory: pytorch, nvidia.
If tensors are placed in pinned memory, it’s also possible to copy them to GPU asynchronously, so that CPU is not blocked by the copying operation. This is useful if for some reason you have CPU-based computation between memcpy and kickstarting forward pass. Enable asynchronous memcpy-to-GPU with your_tensor.to(device, non_blocking=True).
- Documentation on async transfer: pytorch, nvidia.

Notes on parallelization: training

Typical models are composed of sequential layers, so there is not a lot of opportunity to overlap GPU operations. It’s theoretically possible to launch kernels in parallel on the same GPU, but in different CUDA streams, which would benefit graph-like models and maybe even sequential models – this seems to be an ongoing effort in PyTorch.
However, Nvidia Apex library provides optimized DistributedDataParallel which overlaps communication (gradient sharing) and backward pass.
Increasing batch size doesn’t increase parallelism linearly, however there is a fixed cost of memcpy and gradient sharing per batch, so it still makes sense to make the batch as large as possible.
In case of model parallelism, it’s possible to automatically parallelize even sequential models.

Making things faster and avoiding doing things

Suppose you see either CPU, GPU, or network 100% (or almost 100%) used. That saturated resource is the bottleneck i.e. the slowest part for which other resources have to wait. Optimizing the slowest part will speed up the whole thing. Optimizing other parts will have no effect. You can use these tips even if the resource is not 100% used, and then save money by downscaling vertically (reducing number of CPUs/GPUs) without losing training speed.

Making CPU computation faster

To figure out what is so slow in the computation, you need to profile.
- I like py-spy sampling profiler. It can connect to a running Python process, profile both Python and native code, and produce a “flamegraph” visualization where the widest part is the slowest method.
- pip install py-spy if it’s not yet installed.
- Figure out the Python process ID to profile. For the training process, check nvtop to see which process is using GPU. For the dataloading worker process, pick any of them in htop.
- Do py-spy record -r 29 -o profile.svg -p <PID> --native. Wait for 5-10 seconds and Ctrl+C. The resulting profile.svg image is your profile, open it in your browser.
- Caveats:
  - Worker processes are killed and recreated every epoch, so start profiling in the beginning of the epoch.
  - -r argument is how many samples to take per second. If it’s too high, the sampling overhead slows the target Python process down.
  - --native argument will profile even C++ code! But if the profiler just crashes, try to remove --native.
  - --subprocesses will create a combined profile for the main process and all the worker subprocesses, but it’s harder to interpret.
In Parquet, record fields (columns) are written separately from each other, so it’s possible to avoid reading columns you don’t use for training. Petastorm supports that out of the box.
Local caching avoids downloading+parsing+preprocessing the same training example twice. The preprocessed example can be saved to disk and loaded from there during the next epochs. Petastorm supports that out of the box. Random-based augmentation still needs to be done outside of caching!

Some options to speed up slow Python code:
- add @numba.jit decorator to a slow method (even with numpy) for automatic conversion to machine code (there’s more to Numba than that, e.g. parallelization and vectorization)
- memoize computation + parallelize a for-loop with joblib
- manually vectorize code with numpy operations instead of doing for-loops. Make sure your numpy is using MKL!
- parallelize a for-loop with torch.multiprocessing (same as multiprocessing, but uses /dev/shm for communication)
- use Cython to convert your code into optimized C++ and make it an extension
- see if there are different implementations of the same logic. OpenCV can be faster than PIL, etc.
Some CPU computation (e.g. image preprocessing) can be done much faster on GPU instead! We’re working on enabling that with Nvidia DALI library.
You can override the collation function for a faster one, e.g. consider fast_collate from NVIDIA.
Within your training loop, be careful to not use loss variable outside of the loop or use it in such a manner that a reference will be held. If you do that, a separate copy of your model graph will be held in memory, preventing you from using large batch size. See this for more details.

Making network faster

GCS API supports GET-requests that ask only for a range of bytes, but some libraries (gcsfs without configuration) inflate that range to 5MB for caching purposes. The Parquet files don’t benefit from such a large readahead cache (most columns are super small), and this results in unnecessarily huge incoming traffic and a significant slowdown. Check that global variable gcsfs.core.GCSFileSystem.default_block_size is 1024 or less!
Opening and closing connections is pretty expensive. If you need to download a lot of files from GCS, make sure you’re reusing connections.
Again, caching avoids repeated downloads of the same file. Caching could theoretically be done on filesystem level to persist raw bytes (instead of parsed+preprocessed records).

Making GPU computation faster

If you maxed out the GPUs, congratulations! You’re using the hardware efficiently. Still a few optimization opportunities:

Use torch.autograd.profiler to see how long do the GPU operations take. By default, you would only see the CPU’s point of view; with use_cuda=True and num_workers=0, you will see exact GPU kernel timings.
- The resulting profiles are JSON files. Open them in chrome://tracing timeline viewer.
- You might see some kernels, unsupported by GPU, executing on CPU. This is sometimes inefficient because GPU has to send the inputs to CPU and wait for the results to come back. Maybe you can replace it with GPU-supported operations?
- You might also see kernels that are inefficient on GPU and take super long time to execute. You can try placing them on CPU and get speedup even with the overhead of back-and-forth copying!

Try setting torch.backends.cudnn.benchmark = True in the beginning of training. This chooses optimal convolution algorithm if the input shapes don’t change.
“Mixed precision” mode (where some operations work with fp16 inputs but accumulate in fp32) can be several times faster than fully fp32 operations, and also allow you to use 2x larger batch size.
- Try manually converting your inputs to fp16.
- Nvidia Apex library can automatically search for opportunities to enable mixed precision (AMP). imagenet example. PyTorch also has some native AMP support (torch.cuda.amp exists since Feb 2020).
Try reordering tensor dimensions. GPUs like NCHW, but “tensor cores” like NHWC and sizes divisible by 8.
In cloud, you can simply replace a GPU with a more powerful one. Be aware that it can be several times more expensive, but not several times faster!

Closing thoughts

PyTorch is a powerful tool under active development, and it’s useful to learn how it works. Don’t be afraid to profile, experiment, and read the source code!

Road trip across United States

2020-08-14T00:00:00+00:00

This post is the summary of my travel plans, reasoning for it, and some technical details.

The transition to a digital nomad has happened successfully. I’ve moved out of my apartments, stored/threw away/donated my belongings, and left San Francisco. With friends, we stayed a week in Santa Barbara and spent the whole July in a nice tropical-themed house in San Diego. Then the question appeared: where to next?

As you remember from the previous post, my company has said we’re going to be in WFH mode until Labor Day (Sep 7, 2020). Last month, this policy was extended until the end of 2020 with a few clarifications: you can temporarily relocate within the US, but be available during the normal working hours of your team; you’re expected to return to the office in January 2021; and we don’t recommend traveling outside of US due to the possible restrictions at the border. This prompted an idea of a road trip across United States! I’ve been to quite a few US cities in the past years, but there are still many more unexplored ones.

Usually long-distance road trips span several weeks. E.g. traveling coast-to-coast can take about 50 hours or 6 days of 8+-hour non-stop driving each day; but where’s the fun in that? According to the internet, assuming you stay in the points of interest for a few days, a memorable trip takes 3-4 weeks (2-3 “resting” days per each “driving day”). However, in my situation I would have 5 months before I’m summoned back to the office. Of course, I should also work remotely as I’m a full-time employee.

(Side note: actually, this week the company WFH policy was extended again, until the end of July 2021. Google and Facebook also announced their extensions until the same dates a couple weeks earlier.)

So, the plan was hatched: I can start from San Diego, stay each week (or two) in a new place, drive on weekends, work during the weekdays, and be back in San Francisco by January. I’ve reasoned that I can comfortably drive 6 hours per day (this turned out to be pretty accurate). So I can construct a route with the week-long stops up to 12 hours apart (and stay in an intermediate point on Saturday night if needed).

I opened Google Maps and started to pick cities that I’m interested in, never been to, or at least heard something about. I decided I’ll go in the clockwise direction, because the South is too hot in summer, and the North is too cold in winter – we’ll see how that strategy works. Here is my itinerary. There is some buffer, and I don’t need to be in SF in January anymore, so I have quite some flexibility. Please let me know if I should absolutely visit some city that isn’t on the list. You can also let me know that I should absolutely NOT visit some city, but then offer a better one nearby :)

August
- San Diego, California
- Las Vegas, Nevada
- Salt Lake City, Utah (1 week)
- West Yellowstone, Montana
- Casper, Wyoming
- Denver, Colorado (2 weeks)
- Rapid City, South Dakota
- Fargo, North Dakota (1 week)
- Minneapolis, Minnesota
- Milwaukee, Wisconsin (1 week)
September
- St. Louis, Missouri (1 week)
- Indianapolis, Indiana (1 week)
- Cincinnati, Ohio
- Cleveland, Ohio (1 week)
- Upstate New York (1 week)
October
- Boston, Massachusetts (1 week)
- Portland, Maine (1 week)
- New York City (1 week)
- Washington, D.C. (1 week)
November
- Norfolk, Virginia
- Charlotte, North Carolina (1 week)
- Nashville, Tennessee (1 week)
- Birmingham, Alabama
- New Orleans, Louisiana (1 week)
- Houston, Texas
- South Padre Island, Texas (2 weeks)
December
- Austin, Texas (1 week)
- Amarillo, Texas
- Albuquerque, New Mexico (1 week)
- Phoenix, Arizona
- Los Angeles, California (1 week)
- San Francisco

I decided to search the week-long stays on Airbnb, and book the overnight stays on booking.com. On Airbnb, I put in the dates from Sunday to Saturday (or the next Sunday – depends on whether I need to drive both weekend days), filter by “has A/C” and limit the price by $100/night (my inspiration is to fit into $3000/month, which is comparable to what I would be paying for a 1-bedroom apartment, had I stayed in SF). Then I check the location (walkable distance to “civilization” like restaurants and grocery stores), photos, and reviews. For some reason, Airbnb doesn’t include the service fee/cleaning fee in the per-night price, so the lowest-price place might not end up being cheaper!

Another gotcha is non-“instant booking” listings: one time, I submitted a request to book the place, and Airbnb told me that I’ll get a response within 24 hours… except that 24 hours later, there was no response from the host at all, and my request just expired, while I needed a place to sleep tonight. I’m not sure what is the expected behavior here (apart from booking far in advance): if I attempt to book multiple places in parallel, Airbnb warns me I can be double-charged. I’d like to be allowed some redundancy, with the first-response-wins or another strategy.

Of course, to do a road trip, you need a car. In San Diego, I’ve bought a certified pre-owned 2017 Toyota Prius. Thanks to my amazing friends for helping with negotiations – that was a memorable farcical process. Pretty happy with its fuel efficiency: I can travel about 400 miles on a full tank (9 gallons), which costs about $20. Did you know that California has the second most expensive gas in the United States? Only in Hawaii it’s more expensive!

Another concern is the ongoing pandemic. In United States, the number of new daily cases is staying high, and the preventive measures are varying from state to state. Many indoor sightseeings like museums and cathedrals are closed, but many outdoor ones are open and operating. I’m personally trying to stay as safe as possible, wearing face covering, ensuring distance from other people, and so on.

Finally, follow my photoblog: https://instagram.com/petroskoilainen!

Becoming a digital nomad

2020-06-14T00:00:00+00:00

I’m going to move out of my San Francisco apartment and spend some time living in the countryside.

My company (based in SF) has been in working-from-home mode since Mar 9, 2020. The latest communication said that we’ll be WFH until at least Labor Day (Sep 7, 2020). There are doubts about whether it will be considered safe to return to the office at that time. Meanwhile, San Francisco is sinking into its glorious future, becoming less and less pleasant to live in. In the pandemic/quarantine/shelter-in-place situation, it’s clear big cities are overrated. I’ve recently spent a month in Crescent City (small town at the California/Oregon border), working remotely, and enjoyed it a lot. Also I realized that I only need 1 backpack + 1 suitcase worth of stuff to maintain most of my lifestyle indefinitely. So I think I’ll enjoy spending some more time in smaller towns, continuing to work remotely. I’m putting my stuff in the storage and leaving SF for a while.

My stuff
- For the last 5 years, I’ve been living in a half of a 2-bedroom/2-bathroom apartment in SoMa, San Francisco. It’s a pretty new and comfortable building: free gym on the second floor, arriving packages are signed for by the reception and won’t be stolen from your door, if any appliances in your apartment break they’ll fix it for you. I have washer+dryer in unit (a dream of many in SF!), walk-in shower, huge floor-to-ceiling windows, ceramic floor. At the rental price of $2200+utilities, I consider it a great deal.
- The downsides are nitpicks: all the doors are sliding, so sound isolation within the apartment is not great; the ventilation is pretty weird, feels like it’s blowing cold wind all the time, so I had to tape it down. The bigger problem is location: on 9th St and Market St, it’s a pretty rough neighborhood. You’re guaranteed to meet strange-looking people if you walk outside the building in any direction, and you better watch where you step if you care about your shoes being clean. But at the same time, it’s convenient to get to BART/MUNI Civic Center station, Walgreens/Market-on-Market grocery stores, SoMa coffeeshops. The Twitter office is right across the street (the reason I moved there in the first place) - when I worked there, my horizontal commute was shorter than my vertical commute!
- Anyway, over these years I’ve accumulated quite a lot of stuff. Interestingly, mostly things that I got as gifts or planned to give as a gift. Particularly, I collect tech T-shirts as a hobby (I have about 300). Collecting T-shirts is nice because I can show off my collection, one shirt at a time. I have a circular queue rotation system so that I only wear any particular shirt once or twice a year, and a folding/stacking method to store them compactly. But even then, they take quite a lot of space. I also have more buttoned shirts, pants, and jackets that I know what to do with. Besides clothing and gifts, I have some furniture: a couch, a queen bed, a desk, a chair, a floorlamp, a nightstand - not much at all, and most of these things I bought 7 years ago, when I just arrived in US.
- So, I’m going to box most of that stuff and ship it to the storage. I found Boombox Storage where you can rent 5’x10’ storage cell for $135/month, 3 months minimum. They’ll come to you, pick up your packed stuff, and drive it away, then you can pick your items from the online gallery to be delivered back.
- While packing, I’ll go through all the things and see if they spark joy or if I can donate them to Goodwill or throw them away. I did that exercise several years ago (without moving anywhere), got rid of a lot of junk, and established a single place for every category of things.
My address
- It’s useful to have an permanent address (for banks, insurance companies, government agencies). When you don’t have a permanent address, life could be hard. I’ve rented a mailbox at UPS for $30/month, 3 months minimum. This provides you with a real mailing address, not a P.O.Box one. You can come check your mailbox at any time or e-mail them to ask if there is any mail for you.

Overall, I’m pretty excited about the change of scenery, and even about going through all my stuff. See you around!

How to get 100 job offers in two weeks

2018-08-20T00:00:00+00:00

I often get asked how did I choose a job, or how to choose a software engineering jobs in general. I believe the first step is to explore what’s out there and find out what you are worth (arguably it’s useful even if you’re not looking for the job right now). The current reality is that the demand for the engineers is higher than the supply, so naturally it makes sense for companies to reach out to you, instead of you reaching out to them.

Below you will find the list of services that are reflecting this philosophy. They are non-traditional in the sense that they’re not simple job boards where you apply by sending your resume. They provide a higher level of service: either by matching you with companies, or by allowing you to stand out to them. They are free to use for you - they get paid by companies when you’re hired through them.

Disclaimer: I have personally tried all these services (in 2016-2017) and got contacted by companies. Some of the links are referral links. Many of the services will only work with you if you’re looking for job in United States/Bay Area. Some will only work with you if you’re authorized to work in US.

TripleByte conducts their own technical phone screen with you, and then matches you with companies. If you’re interested in any of those companies, you can go directly to onsite interview. I think this is a nice approach that allows you to skip repetitive phone screens. They supposedly also help you to negotiate your offer. Last time I used it, I’ve got about 6 hand-picked matches.

These two have a selection of partner companies. You choose a company, solve their coding challenge online, and then the company contacts you.

HackerRank Jobs is a programming competition website. They run competitions sponsored by employers so that the sponsor can contact the top performers. This is how I got my first full-time job in US! However, they also have a section where you can apply to companies by solving their coding test. No competition participation needed.
CodeSignal is also a tournament website. They have a “Company Challenges” section with several companies, where they claim that “solving these challenges will increase your chances of getting matched to open positions”. They also have a “Jobs” section where you can create a profile and get matched with companies.

There are many services where you fill out the profile, they put you in an “auction” where companies “bid” on you, and you decide which ones to interview with. This is really reversing the job search. The auction is usually limited-time. The best part is that companies make you “pre-offers” with salary and equity, so you don’t waste time if those numbers are too low. However, it does require some investment on your side to fill out and polish your profile (specify your experience, desired industry, technologies, etc.) - but I believe it’s worth it.

Hired probably has been there the longest of them all. They have a lot of small (1-15 employees) companies. Last time I tried it, I’ve got 37 pre-offers in two weeks.
Indeed Prime is a pretty similar service. About 27 pre-offers in two weeks.
A-List showcases you to startups on AngelList.
Vettery is based in NY and their companies are mostly from NY (or at least that was my impression). About 20 pre-offers in two weeks.

Some services are similar to the above: they let you fill out the profile and then companies send you pre-offers - but the “auction” is not time-limited. However, I find that the interest to your profile is only high for a few weeks.

Gloat
Woo.io. About 10 matches in the most active month.
Underdog. About 6 matches in the first week.
Anthology

All in all, during my last job search, I got contacted by about 100 companies (not including the ones reaching out through e-mail or LinkedIn). Your mileage may vary! I talked to about 25, went to 17 onsites, got 7 offers, and accepted one :) Good luck!

Thoughts on engineering process

2018-08-02T00:00:00+00:00

Engineering is solving problems. Software engineering is solving problems with software. Professional SWE is doing so sustainably, repeatedly, continuously. This means some kind of process is necessary.

Understand the problem

It starts with problems. It’s very rare to see someone asking: what problem are you trying to solve? I personally ask it all the time. What are the goals we want to achieve? What usecases do we expect? Understanding the problem makes the next steps much easier, and you eliminate a lot of failure cases. It’s very cheap to iterate on problem statement. It’s very expensive to jump to implementation and at the end, realize that the problem was underspecified, or that you were solving the wrong problem all this time. This applies both to high-level goals as well as to day-to-day tasks. Of course, the cost of mistake is higher in the former case.

When someone tells you to do something, and especially says it’s urgent, it’s tempting to just jump into coding. Stop. Are you sure you understand what problem they want to solve? Separate what they want to achieve (the problem) from what they tell you to do (the “solution”). Engage your critical thinking. It will often be the case that the actual thing that needs to be done is very different from what they told you originally.

Similarly, when someone asks you how to do something, it’s tempting to just offer an advice. Stop. Are you sure you understand what problem they are trying to solve? Separate what they want to do (the problem) from what they ask you how to do (the “solution”). It’s called the XY problem. It will often be the case that you need that context to provide better advice.

Clarify the requirements ruthlessly. Get them to open a ticket. Bonus points if you can agree on metrics to define how do you know when the problem is solved.

Design before implementation

When you understand the problem, it’s tempting to jump straight to the implementation. Stop. Think about your approach, on a high level. How can it fail? Again, engage your critical thinking: you can foresee a lot of failure cases. It’s possible that at this point you’ll realize the problem is underspecified - it’s great, go back and clarify the requirements. Consider different approaches and compare them. A little imagination goes a long way. Again, it’s cheap to iterate on the design. It’s very expensive to jump to implementation and at the end, realize that there already exists a solution, or there is a better approach. This applies both to large long-term projects as well as day-to-day ones. Of course, the cost of mistake is higher in the former case.

Describe your approach. For larger projects, write a design doc. Iterate on it. Ask others to review it. State the motivation, goals, non-goals, limitations, and risks. Come up with alternative solutions and compare them.

If you think hard and make important design decisions at the design stage, you’ll notice that the implementation actually becomes pretty much straightforward. The explicitly written problem statement and the high-level design document will make the code review much easier. They will also be helpful when someone else many months later will try to understand why things are implemented the way they are.

Bottom line

For smaller tasks, the temptation to shortcut the process is higher. It’s understandable; at a minimum, use the ticket to iterate on the problem statement and the approach. At the end of the day, it’s the result that counts. And the result is that you’ll be solving exactly the problem you want to solve, using the most suitable approach that you chose consciously.

Always ask what problem are we actually trying to solve. Don’t waste time solving the wrong problem. Don’t overengineer. Think about failure cases. Ship reliable software.

A case for written communication

2018-07-21T00:00:00+00:00

TL;DR: In corporate world, explicit written communication is better than word-of-mouth and telephone game. Most meetings are better when they start and finish with a written document. To become a happy and productive team, ask to put things in writing.

Human memory vs. written documents

One day you might notice that none of your team meetings have any writing associated with it. It’s almost like every meeting you have has no agenda in the beginning and no action items at the end - except they most often do, but that information is all “remembered” - stored in people’s memories.

Human memory is a bad storage. Having to remember too many things induces stress. Accessing memorized information needs the human to be alive, awake, and employed. It decays over time. It’s barely searchable. There is no way of making sure it’s synchronized (as in, my memory is the same as yours). The only way of sharing it is through speaking. Speaking is a lossy protocol: you said one thing, I heard another, and remembered a third. Speaking is synchronous: it requires ongoing attention from all participants. Speaking is non-repeatable: what was said once has to be repeated for another person at the same cost.

On the other hand, written document is great storage. Information there is preserved after layoffs. It’s trivial to have a single version of it. It never decays. It’s searchable. After single-time cost of writing, it can be shared in its entirety for free. Writing is lossy as well, but the chance of miscommunication is lower. It’s asynchronous: readers can consume it whenever it’s convenient. Written document can be shared with more readers later. Moreover, it can be evolved over time, collaboratively. For most people, typing is as fast as speaking and reading is faster than listening.

When you should prefer writing

Spoken communication is great for synchronous bidirectional interactions: 1:1s, brainstorms, status updates, pair programming. It’s great for hackathon-style development: small team with short-term goals, all physically in the same room. But there is no excuse not to have something in writing when the communication involves many people and potentially more later, or when the information is important to get right, or when the information is expected to have a long lifetime and be referred to repeatedly, or when it requires feedback from everyone but not immediately: planning on monthly/weekly scale, requirements, design documents, APIs.

Writing is great tool for meetings: it encourages coherent thought, things become explicit, hidden problems bubble up and provoke questions from co-authors and readers both. Why would you use a sub-optimal tool when there is a better one? If you’re holding a meeting, at least sketch an agenda; it will structure the discussion and limit misunderstanding. During the meeting, flesh out the agenda items as you go through them and make decisions. Use the resulting document for the next meeting in series. Bonus points if you can simply share the document and ask for feedback instead of holding a meeting. Any meeting that did not result in any written document means time spent sub-optimally.

Push for more things stated in writing. If someone is hesitant to do so, they value neither their time nor yours. They want to hoard tribal knowledge in their heads. They don’t want you to be productive in the long term. They don’t want to encourage feedback. They want to perpetuate hackathon culture. They want to make your life stressful.

What is compensation made of

2018-06-03T00:00:00+00:00

Disclaimer: This post is about software engineer compensation in United States. I’m using 40% total tax rate in calculations (can be lower depending on your family situation). I am also not a lawyer or tax advisor.

Sometimes you find yourself in the situation where you have multiple job offers and need to choose between them. There are many variables that should come into your decision, and one of them is comparing the financial side of the offers. Some of the components are not obvious, so this post will hopefully help you to evaluate the total fair market value of your offers. Make sure to ask your contacts at the company about these items, if they are not explicitly stated in writing.

Basics

Base yearly salary. Pretty clear. Usually paid every other week. For Bay Area, varies within $100k-200k for most people, depending on experience. Taxed at about 38% (22% federal tax + 8% California tax + 6% Social Security tax + 2% other taxes).
Equity: RSU/options. There are multiple articles explaining how to evaluate equity, so I won’t go into details. RSU profits are taxed a bit higher than cash salary. Usually equity is vesting over four years, so total yearly compensation is calculated as base_salary + equity_grant / 4. However, there are other components in the equation.

Bonuses

Signing bonus. One-time taxable bonus, usually is given with the first paycheck. Can be as high as $100,000 (e.g., for returning interns at Facebook). If you leave the company before your first anniversary, you would be asked to return it pro-rated for the rest of the year. Can be easier to negotiate than the base salary: for the company it’s a one-time expense, as opposed to a recurring one.
Relocation bonus. Also taxable and usually given with the first paycheck (which means you will need to spend your own money until then). Can also be non-cash: the company might pay for the ticket and provide accommodation e.g. for the first month.
Annual bonus policy. Profitable companies often have a “target” annual bonus which can be 10% or 15% of the base salary, and the actual bonus for the year is somehow computed from the company performance and your personal performance.
Raises/promotion policy. Usually the companies at least try to keep up with inflation, so you can expect 1-2% raise yearly. However, you should also keep in mind whether it’s going to be easy to get promoted (and get a raise/bonus associated with it).
Performance/per-project bonus policy. Some companies reward engineers based on evidence they’ve put in good work: after a performance review, or after a release was shipped.

Bonuses might be taxed a bit higher than your normal pay.

Other perks

ESPP. You can set off some post-tax money from your paycheck to buy your company stock at a discount, e.g. 15%. The amount that can go towards the purchase is limited by the Purchase Plan at some % (e.g. again 15%) of your base salary, and by IRS at $25,000 per calendar year - whichever is lower - so the minimum profit (if you sell immediately) is 0.15 x min(25000, <base>x0.15), which is 2.25% of your base or $3,750, whichever is smaller. The better the discount and the better your stock is doing, the better your profit.
PTO/sick leave policy. There are 50 full business weeks in a year. If two companies offer the same base pay, but company A offers 15 days (3 weeks) of vacation, and company B offers 20 days (4 weeks), that extra week is equivalent to getting 1/50 = 2% of the base pay without having to work for it. You basically get paid extra 2% for every extra vacation week over baseline. Be cautious about the “unlimited PTO” policy: as you can guess, it’s not really unlimited - either you come up with a guilt-induced limit for yourself, or you hit the informal unspoken limit, beyond which your manager is not comfortable with your absence.
401(k) matching. Some companies encourage you to set aside a portion of your paycheck (pre-tax) to save for your retirement, by matching your contribution to your retirement account. If the company has this program (not just 401(k) program, but 401(k) matching program), then for every dollar you contribute, the company will contribute 50 cents (if they match 50%) or a dollar (if they match 100%). There is a limit set by government on how much you can contribute per year ($18,500 as of 2018). Additionally, the company may have a limit of how much they will contribute (e.g. $1,500/year at Twitter). The company’s one-time contribution happens after the year has ended, e.g. in mid-February of the next year.

Reimbursements and subsidies

Free/subsidized food. In Bay Area, this is easily worth $20-$30 per meal, $50-60 with dinners. $25 daily (post-tax) = $10500/year (pre-tax).
Shuttles/commuter subsidy/Uber credits. Depending on how you commute, every $12 (post-tax) spent on a daily commute are worth $5000/year (pre-tax).
Reimbursement policy: books, conferences, travel, offsites. This could be easily worth thousands per year.
Gym reimbursement. In Bay Area, this can cost $50-150 per month (post-tax), so gym reimbursement is equivalent to $600-1800/year (post-tax) or $1000-3000/year (pre-tax).
Insurance premiums for you and your family. This is $200-300 per month = $2400-3600/year. However, almost every company takes care of that for you, so it’s unlikely to be a decision factor.

Other concerns

If you’re choosing between different states, take into account:

state taxes (varies from 0% to 13%)
cost of living (especially rent - expect to spend $2000-$3000/month (post-tax) = $40k-60k (pre-tax) in Bay Area)
sales tax (varies within 6-9%)

This should help you to put a price tag on many perks and benefits that the companies do or don’t provide. Remember that it may be short-sighted to compare companies from the financial side only, and also to evaluate offers based on the current stock price or current startup valuation. Who knows what will happen in a couple years? Good luck!

How to call Scala UDFs from PySpark

2017-02-17T00:00:00+00:00

Motivation

Suppose we decided to speed up our PySpark job. One possible way to this is to write a Scala UDF. If we implement it in Scala instead of Python, the Spark workers will execute the computation themselves rather than ask Python code to do it, and won’t need to serialize/deserialize the data to the Python component. Double win!

So, we can write a simple Scala object with a single function in it. Then we wrap the function in udf. I sincerely apologize for the imperative style.

package com.example.spark.udfs

import org.apache.spark.sql.expressions.UserDefinedFunction
import org.apache.spark.sql.functions.udf

object Primes {
  def isPrime(x: Int): Boolean = {
    if (x < 2) return false
    var i = 2
    while (i * i <= x) {
      if (x % i == 0) {
        return false
      }
      i += 1
    }
    return true
  }

  def isPrimeUDF: UserDefinedFunction = udf { x: Int => isPrime(x) }
}

Save it as src/com/example/spark/udfs/Primes.scala or under any other convenient name.

Compiling Scala

We don’t need heavyweight IDEs or sbt to compile such a simple function. Let’s do it from the command line! We would need to use Scala 2.11 when compiling (as of Spark 2.0.1/2.1.0). Download Scala 2.11 and unpack it somewhere (I unpacked it as ~/Downloads/Distribs/scala-2.11.8).

Setup the environment variables:

SCALA_HOME=$HOME/Downloads/Distribs/scala-2.11.8
PATH=$PATH:$SCALA_HOME/bin

Make sure we also have $SPARK_HOME set:

$ echo $SPARK_HOME
/Users/sserebryakov/spark-2.0.1-bin-hadoop2.7

Check that we can invoke scalac:

$ scalac -version
Scala compiler version 2.11.8 -- Copyright 2002-2016, LAMP/EPFL

Go to our project folder and build a JAR. Note that the wildcard is just a star, not *.jar:

$ scalac -classpath "$SPARK_HOME/jars/*" src/com/example/spark/udfs/Primes.scala -d primes_udf.jar

Running PySpark

Now we can provide the JAR in the classpath for pyspark:

$ pyspark --jars primes_udf.jar

Now we can use the Scala function like this. The prerequisite is that the data should already be in the DataFrame format. Be careful with the syntax, the py4j exceptions are rarely helpful.

from pyspark.sql import Row
from pyspark.sql.column import Column, _to_java_column, _to_seq

a = range(10)
df = sc.parallelize(a).map(lambda x: Row(number=x)).toDF()

scala_udf_is_prime = sc._jvm.com.example.spark.udfs.Primes.isPrimeUDF()
is_prime_column = lambda col: Column(scala_udf_is_prime.apply(_to_seq(sc, [col], _to_java_column)))
df.withColumn('is_prime', is_prime_column(df['number'])).show()

This will print:

+------+--------+
|number|is_prime|
+------+--------+
|     0|   false|
|     1|   false|
|     2|    true|
|     3|    true|
|     4|   false|
|     5|    true|
|     6|   false|
|     7|    true|
|     8|   false|
|     9|   false|
+------+--------+

Tested with Spark 2.0.1, but should work for 2.1.0 (the latest as of this writing) as well.

Currying

If we want to initialize the Scala class with some arguments once, and call the function repeatedly, we can use a case class. Example:

package com.example.spark.udfs

import org.apache.spark.sql.expressions.UserDefinedFunction
import org.apache.spark.sql.functions.udf

case class DistanceComputer(originX: Double, originY: Double) {
  def distanceUDF: UserDefinedFunction = udf { (x: Double, y: Double) =>
    math.sqrt(math.pow(x - originX, 2) + math.pow(y - originY, 2))
  }
}

Registering with PySpark would look like:

scala_udf_distance = sc._jvm.com.example.spark.udfs.DistanceComputer(0.0, 0.0).distanceUDF()

Make sure to conform exactly to the argument types when calling the constructor. A constructor with Double argument should be called with 0.0, not 0.