Which tools do you use for analytics projects? [Thread]

OS

  • Mac OS Sierra MBP i5 2.5 w/16 gigs ram
  • Used to use Linux (Elementary OS because I'm fussy)

Text Editor/IDE

  • RStudio for all my R stuffs
  • Rodeo by yHat for Python (but "it's been awhile" sung in Stained voice)
  • I used to live in Jupyter but if I have to work with an .ipynb file I prefer nteract

Terminal

  • iTerm2
  • ohmyzsh (ZSH_THEME="agnoster")
  • vim (for quick notes exploring in terminal)
  • MacPorts (I know almost everyone uses Homebrew but coming from Linux Macports seemed more familiar and it has served me well!)

Version Control

  • git
  • GitHub
  • hub (GitHub cli)

Messaging

  • AIM
  • Slack
  • Tweetbot

Analysis notes

  • I don't really use .R files, I'm pretty much all in on the .Rmd workflow and document outside of code chunks with markdown. I really like using tint a lot.

Misc

  • clamscan for anti-virus
  • Wunderlist so I don't forget

With Rstudio I really like to keep the right side closed as much as possible; I only open it when I need to look something up, e.g., ?quos(). The notebook workflow is life :raised_hands:t2:

2 Likes

Really don't use Excel (that's a lie, but like 99% of the time I can avoid it).

Excel pales in comparison to R for modifying CSV files. I actively make a fuss at work when my teammates pass around Excel files. Don't get me wrong, Excel is useful tool and is very often how people start off interacting with data to make decisions. BUT, analysis in Excel is extremely prone to user error and irreproducible results (think cell reference issues, typos, pasting values over formulas).

In my experience, it's one of those things that you realize how terrible it is only after it bites you. Once you have to pass a complex Excel file back and forth between people and explain why deleting cells referenced elsewhere just destroyed the entire thing... yeah, no thanks, never again.

#rmarkdown4life

2 Likes

Yes, love the powerline plugin that comes with agnoster theme!

@jakekaupp

I have a love/(mostly)hate relationship with Papers. Last year I started using https://paperpile.com and like it a lot. It's not Papers and doesn't really have an offline mode (besides syncing your google drive) but I've only opened Papers twice in the last year.

Maybe Zotero is good? Once I year I would do the rounds and see if I could stomach the alternatives (Zotero, Mendeley, etc.). None of them really did it for me.

Papers seemed like a classic case of really bad QA and features that nobody cared about which wrecked the important stuff (my $0.02).

1 Like

OS

  • Win10 - haters, feel free to hate

Text editor/IDE

  • RStudio like 90% of the time…
  • Atom!
    • Monokai theme is :fire:

Terminal

  • Good ol' fashioned cmd

Version Control

  • GitHub from cmd line
  • GitLab from cmd line

Messaging

  • Slack
  • Hangouts

Analysis notes

  • rnotebooks and md docs (occasionally built into blog posts via blogdown when I'm not procrastinating on updating it :slightly_frowning_face:
  • 3.5" x 5.5" notebook (Grid layout) that I have dubbed my brains (pic below)

Project Management

  • Trello

My Brains

1 Like

This is very interesting! It's great to see other tools people are using.

OS

  • Windows 7 (on a trusty Lenovo Thinkpad)

Text editor

  • Rstudio - Of course. With the default theme and font
  • VSCode for anything else, eg Markdown, SQL (for intellisense when working with Azure SQL server)

Command line

  • Powershell - For the very small amount I do on the command line. This is more driven by the rest of my team using it.

Version control

  • Git and Github for my personal projects
  • VSTS for work (Not data related projects)

Messaging

  • Email
  • Slack - This has been fantastic for creating a sense of team with semi remote members. Also very useful way to ask questions and broadcast information to a group

Analysis notes

  • R Markdown or Notebooks

Misc

  • Blogdown - For displaying some of my personal work and thoughts. This is a fantastic tool that I'm still getting getting to grips with

Things I am curious to start using

  • Docker - I ran my first Rocker/Tidyverse container the other day! I'm not sure my use warrants it yet, but I still appreciate the magic of it.
  • D3.js for creating interactive visualisations
1 Like

The replies aren't quite what I thought they would be when I read the title. So here's an attempt to derail the thread.

Here's how I R;

Rmd to .docx or .pdf for documents
Rmd to beamer with corporate theme then ghostscript to individual slides images to powerpoint because conference organisers won't accept anything else

  • dplyr & broom to pixiedust for tables output.
  • ggplot & cowplot & Cairo for graphics output (Cairo to add antialiasing)

  • two personal packages - one for misc stuff, mostly wrappers to make inline R nicer. One for working with data from a specific source.
  • lme4 for mixed-effect regression models.
  • most of the tidyverse to make life easier
  • openxlsx for when I'm obliged to write data to Excel
  • RODBCext for database connections
  • janitor for cross tabbing during eda.
1 Like

@Max I have been using that app from the beta version, and while there are issues, there still isn't anything in the general vicinity that met my needs. I guess I'm so blind to it now that I don't really see the flaws, but I do admit the bad QA (the downfall of all academically generated software).

Personally, I have to be flexible though as we write with many collaborators so I'll use nearly every citation format and manager just to make sure to make things easier. Zotero is usually the manager of choice amongst most, but I still run into the 'Whats a citation manager?'. It's 2017 and there are people that still paint in caves....

So much nerdage in this thread, love it!

OS

Work: Win10
Home: Ubuntu

IDE / Text Editor

RStudio
Atom
vim

Other Technologies

SAS (an unfortunate fact of life SAS)
SPSS
TreeAge
MSOffice

Command Line

Terminal

Version control

git / GitHub

Messaging

Email and Telegram

Analysis Notes

So. Many. Post-its! Oh and also my version of BRAINS!!:

Project Management

Trello for myself, Asana for the rest of my team

Reporting

LaTeX (I love TeXnicCenter)
Word

3 Likes

OS

  • Sierra on 1st gen rMBP (15"/i7/16Gb/250Gb) circa 2012, still going strong

Terminal accoutrements

  • iTerm2 (I didn't see the difference from Terminal at first, but now I'm hooked)
  • bash with a ridiculously customized prompt, though this thread has made me consider zsh
  • homebrew (w/cask) to install and update everything (except R...)
  • gcc-8 installed with homebrew to compile everything, but I'm pretty sure this is why I keep crashing R so much

Editors

  • RStudio for everything R
  • PyCharm/Jupyter for Python
  • Sublime/Atom are overloaded with plugins that need to go, so
  • TextWrangler for viewing/editing when Sublime/Atom are too slow
  • MacDown for Markdown

VC

  • git/GitHub via
    • RStudio where possible,
    • the command line if I'm feeling competent, and
    • the GitHub frontend when I'm not

Other necessities

  • Flycut for a super-light multi-clipboard with history. My favorite and possibly most-used app ever; it just does its job perfectly and without intrusion.
  • Quicksilver, because it's still better than Spotlight
  • TweetDeck for Twitter, which downloaded 7Gb of images onto my computer and hogs memory, and which is thus a candidate for replacement
  • DaisyDisk, a beautiful interface for figuring out when TweetDeck is using 7Gb of storage
  • Firefox because I can't do without tabs of tabs via Tab Group Helper, though I'm fond of Vivaldi
  • OnyX for cleaning up
2 Likes

YES :fist:t2:

powerline is life; also solarized dark is something I keep returning to for some reason as well. I also use the Roboto Mono for powerline font, which I'm really liking lately!

Well this is fun!

OS

  • Sierra on MBP (17", i7, 16GB RAM), mid 2015
  • Windows 10 (Lenovo Thinkpad, i7, 16 GB RAM, etc.)

Messaging

  • Franz - keeps all of my messaging in one place (Hipchat, Slack, Skype, Messenger) :grinning:

iDE / Editors

Terminal

  • Hyper - really my all time favorite!
  • Terminal - when I forget to use hyper

Analysis Other than R

  • Tableau - basic BI work for some dashboards requested by execs
  • Excel - still a great calculator and presentation

Version Control

  • Github
  • Bitbucket

Other Site Services

  • Dropbox
  • Google Cloud Platform - If you need more computing power, here is the place to go
  • Digital Ocean - depending on various projects, I sometimes need extra resources and throw up an instance into the cloud (i.e. demonstration of a shiny app)

Resource Management

  • Papers - find this to be the best way to keep track of various academic papers
  • Evernote - my notebook for anything and everything as it integrates with so many different things
  • Google Suite - Does everything you really ever need it to do

Productivity

  • TextExpander - after creating my own shortcuts it drives my coding productivity and fulfills any of my needs and syncs between both of my machines
  • Alfred - favorite productivity bar on the Mac
  • Keyboard Maestro - Newer adoption so I am still working with it and getting it to what I need it to be
  • Todoist - cheap for the pro edition and gives me so much flexibility
  • Feedly - Blog aggregation
  • Tweetbot - I like to read and see whats happening in the world
2 Likes

I felt compelled to jump in. I'm in social science, usually working with human subjects data but occasionally text and demographic data; all for academic research.

OS

  • macOS (Macbook Air 13")

Text Editors

  • For R, always RStudio. I'm always looking for new IDE toys to play with, but nothing comes remotely close to RStudio from what I can tell.
  • For quick work or opening unfamiliar/txt files, I like CotEditor for its balance of features and speed.
  • For more involved work in multi-file non-R projects, either Sublime Text 3 or Atom depending on how the wind moves me. I use Atom for managing my Hugo site.
  • I like Texpad for LaTeX work, though it's not leaps and bounds ahead of the FOSS alternatives and has some limitations.
  • I have Rodeo on my computer ostensibly for Python, but the last time I needed to work in Python (which is not really my thing) I ended up using RStudio with the new Terminal feature to get it done. Made the workflow easier as I didn't want to do data manipulation in Python where I'm less comfortable, just wanted to pass the formatted data to the Python-only library I needed and then get it back into R before I broke something.

Command line

  • iTerm 2 with some light theme tweaking

Version control

  • git, usually via GitKraken

File browser

  • Path Finder

Messaging

  • Email for colleagues
  • Telegram for close friends/family

Analysis Notes

  • I've gradually moved to having my analyses living in Rmd files (short of the raw data, of course). I can share results with colleagues by knitting and if anyone wants to audit the work, the entire project can be rebuilt from scratch in a way that is totally transparent and well-documented (mostly).
  • When writing manuscripts, it is almost always MS Word. Most colleagues are not R users and even less LaTeX users. MS Word really does add value with its change tracking and commenting which are not easily replicated via something like a git repo w/ LaTeX.

Misc

  • In the process of changing from LastPass to BitWarden for password management. BW lacks secure notes and form fills, which makes it a partial switch for now. LP was killing my browser performance, so I had to get something else for Firefox.
  • Zotero (with a Zotfile-aided workflow) is central to both the organization of research articles and citation in manuscripts.
  • For bigger jobs or long-running analyses, I fire up an Amazon EC2 instance with RStudio Server and run it there.
  • I've switched to using Brew to manage my R installation. I found that by using source-compiled R and R packages, many operations are much, much faster.
  • I use Skim as my PDF viewer, despite RStudio's insistence on overriding my default and opening them in Preview.
  • Bartender unclutters my menu bar.
1 Like

@Ranae just a mention that you can also access the command line from RStudio now, in case you need it :blush:

1 Like

Ok, so, since atom's been crashing on me regularly, I gotta ask: is Sublime really $80 worth of awesome?! @cdr6934, @terence, @duythonguyen, @dylanjm, @jessemaegan…other Sublime fans— any words to the wise on this?

2 Likes

I understand the pain @mara

I purchased Sublime before Atom came out. At the time, once Atom came out I was excited as there were few updates occurring to Sublime and some of the community was worried about the abandonment of Sublime. But after a couple weeks, I found a few things:

  • it would crash on me using certain plugins which I have yet to have happen with Sublime (on both the mac and pc).
  • Opening 100+ MB text files and doing some regex manipulation to the files would work with no problems in Sublime, but in Atom (at least at the time), it could not bear it and just whithered.
  • On the PC, my machine will restart due to automatic updates (which I can't control), Sublime opens up to where I was previously with no lost data. I have yet to find a program that does it as well as Sublime.

So after a while, I just went back to Sublime and haven't looked back to Atom. Now that I have a library of custom packages built for my workflow, it would be some conversion cost for me to do so even if I would try Atom again as my go-to text editor outside of the terminal.

2 Likes

While Atom has grown on me, I'm an emphatic yes for Sublime!

3 Likes

You can download a full featured, no-nag copy of Sublime from the Sublime site, try it as much as you want, if you don't think it is worth $80 all they ask is that you stop using it.

4 Likes

Ugh, yes!! @jessemaegan, @danr, I think you guys have sold me! Even after uninstalling most atom packages it's still being wonky, so it's time for a change!

3 Likes

Hi @mara, Sublime is much more stable and faster than Atom, in my experience. I tested Sublime, Notepad++ and Atom for a long time, and Atom is by far the less stable of the three. If a free license is a must-have, I would go with Notepad++, otherwise Sublime 3 is the best of the bunch, and as noted by @danr you can test it for as long as you need in order to decide if it's worth the 80 $.

One final note concerning plugins: Atom has a reputation of being the most customizable and having the most plugins, but I didn't find Sublime fall short on the quantity of plugins. It even has a Python IDE (!) (not very aptly named Anaconda, though it has nothing to do with CA Anaconda) with code completion, tooltips, docstring opening, jump to method definition, linting and others. Of course, no such plugin is needed for R, because RStudio is the best IDE :grinning:

2 Likes