Blog


Subscribe to posts

Workflows in bioinformatics

posted Jan 23, 2015, 2:40 PM by Klc Kelsey   [ updated Jan 23, 2015, 2:51 PM by Thomas Sierocinski ]




Was following the chatter on twitter regarding the E-InfraMPS2015. 

"The E-inframMPS2015 workshop brought together scientists sharing their experiences on how to build efficient and sustainable e-infrastructures for massively parallel sequencing data management and storage, as well as setting up and maintaining an associated ecosystem of workflows, pipelines, and bioinformatics software."

There was much information to digest at E-inframMPS2015. In the post i'll be focusing on summarizing the ecosystem of workflows in bioinformatics. 






Chipster



Chipster provides easy access to >340 analysis tools, no programming/command-line experience req'd. Free, open source.
Visualize in GUI, share analysis sessions, integrate data. Java Web Start application. #einframps2015
Chipster easy to install. VM contains all the analysis tools and reference data. Easy to add new tools.




Bpipe





Bpipe. "pipeline interpreter and runner", written in Java and Groovy.


Bpipe has good audit trails. Runs on all major queuing systems.


http://www.ncbi.nlm.nih.gov/pubmed/22500002


piper



Johan Dahlberg on the Piper workflow system built on top of GATK Queue
Queue: parallelizes workflows, robust (reruns failed jobs), traceable, deletes intermediate files, reusable components


Queue is "battle-hardened" (has been used a lot by e g Broad Inst.), has nifty utilities for NGS.




https://www.broadinstitute.org/gatk/guide/topic?name=queue








A pipeline project started at the SNP&SEQ Technology platform built on top of GATK Queue. Since then Piper has been adopted by the Swedish National Genomics Infrastructure (NGI) for use in the the Swedish Genomes Program as well as for samples submitted through the Illumina Genome Network to the NGI platform.

Piper builds on the concept of standardized workflows for different next-generation sequencing applications. At the moment Piper supports the following workflows:
WholeGenome: For human whole genome sequencing data. This goes through alignment, alignment quality control, data processing, variant calling, and variant filtration according to thebest practice recommended by the Broad Institute, using primarily the GATK.
Exome: TruSeq and SureSelect human exome sequencing: These use basically the same pipeline as the whole genome pipeline, but with the modifications suggested in the best practice document for exome studies.
Haloplex: Haloplex targeted sequencing analysis. Including alignment, data processing, and variant calling.
RNACounts: Produces FPKMs for transcripts of an existing reference annotation using Tophat for mapping and Cufflinks to produce the FPKMs.

All supported workflows are available in the workflows directory in the project root.


Hitachi


Johan Westin from Hitachi Data Systems on Hitachi Object Based solutions for genomics




Luigi



Samuel Lampa (BILS/Dept of Pharm. Biosci. in Uppsala) on using Luigi, a framework developed by @spotify, for bioinformatics


Luigi supports both command line and Hadoop execution. Powers 1000s of jobs each day at Spotify.


Luigi has a nice visualization interface for task execution. Shows dependency graph.
Luigi is task focused. Implement requires() and output() for each task. This implicitly defines a dependency graph.








CRS4 on Galaxy

Luca Pireddu, CRS4 on Galaxy as a workflow manager. Execute and record operations on data.
Galaxy's web interface not suitable for automation, so control it using its REST API. Adapted BioBlend to this end
Created the "Hadoop-Galaxy adapter" to address incompatibilities between Hadoop and Galaxy #einframps2015


Collections of tools for galaxy to incorperate hadoop and other hadoop based tools (crs4?)


SnakeMake


Maciej Kandula (Chair of Bioinformatics Vienna) on Snakemake
Lightweight workflow systems. make is great but lacks some features. Snakemake and Bpipe are similar in many ways.
Recommends Sean Davis' snakemake tutorial http://watson.nci.nih.gov/~sdavis/blog/flexible_bioinformatics_pipelines_with_snakemake/ …






CloudGene



Sebastian Schönherr from Medical U of Innsbruck in Austria on Hadoop pipelines for NGS analysis
CloudGene executes MapReduce prgs and combines them into workflows. Runs in your browser. Integrates existing tools.



Built around the idea that Linux is the lingua franca of data science.



Stream oriented

Nextflow extends the Unix pipes model with a fluent DSL, allowing you to handle complex stream interactions easily.
It promotes a programming approach, based on functional composition, that results in resilient and easily reproducible pipelines.

Fast prototyping

Nextflow allows you to write a computational pipeline by making it simpler to put together many different tasks.
You may reuse your existing scripts and tools and you don't need to learn a new language or API to start using it.

Unified parallelism

Nextflow is based on the dataflow programming model which greatly simplifies writing complex distributed pipelines.
Parallelisation is implicitly defined by the processes input and output declarations. The resulting applications are inherently parallel and can scale-up or scale-out, transparently, without having to adapt to a specific platform architecture.

Portable

Nextflow provides an abstraction layer between your pipeline's logic and the execution layer, so that it can be executed on multiple platforms without it changing.
It provides out of the box executors for SGE, LSF, SLURM and PBS/Torque batch schedulers and for the DNAnexus cloud platform.

Continuous checkpoints

All the intermediate results produced during the pipeline execution are automatically tracked. This allows you to resume its execution, from the last successfully executed step, no matter what the reason was for it stopping.

Reproducibility

Nextflow supports Docker container technology. This, with the ability to run on multiple platforms, allows you to write self-contained and truly reproducible pipelines.










Nexus 4 Review - Feels great in the hand but will probably break in the bush!

posted Apr 26, 2013, 4:49 PM by Klc Kelsey   [ updated Apr 30, 2013, 3:07 PM by Thomas Sierocinski ]

The Beginning:

With great reservation i ordered the Nexus 4.  
I wanted Android. 
I wanted updates from Google direct.  The Nexus Brand
A 2nd hand galaxy nexus was cheap, locked but easily rooted. 
The nexus 4 was about the same price,direct from Google and a write off (developer expense). 


I'm not great at commitment. 
And contracts with Telcos are beyond me. 
It needs to be said that i'm a 2nd hand phone queen.
I've been  buying phones (2nd hand on craigslist, nvr pay more than 3 bills)  and try not to break them (which is how it will die and then i will buy another).  I'm tuff on tech =p  Its there to be used, abused, loved and then thrown away for the new model. 

I rarely buy new, but i bought a new Nexus 4.

Started off ugly.  What a gong show via Google. 
Years previous I had signed up for an android developer acct and was getting news of the nexus 4. 
Google had even given me a specific time and day I was going to be "allowed" to purchase one (10:40pst Nov 12/12). 

Well come allotted day and time -  Google Play Store is totally broken/overwhelmed by demand. 
I managed to complete a "check out" once only to get a email saying sorry try again next week.   
Oh snap!

Long story short:

Had phone in my hands just after x-mas, Google took my money in early November. 
16 gigs - $420 Canadian buckaroos after taxes and shipping. 

Review:


Screen 

4.7" screen.  I was worried about this.  I mean shit, is that gonna fit in my pocket.  I'm no hipster with skinny jeans and no where to put my phone.  If i need a purse to carry that shit around then by definition this is not a "mobile phone".  

Size

Good news... Fits in most pockets thus far.  Its not that it doesn't fit, the question is will it fall out?
Thus far no but... my PJ pockets and my hoodie pockets are death traps for this phone.  
As it kinda not really fits in those sub-sized loose pockets and would definitely be going for a beater if i put it in there. 

Also as a someone that's averagely built this phone is towards the max.  My hand does fit, i can do one handed operations but i worry about dropping it. I wouldn't want this phone if i was small handed.  Really need to have phones come in different sizes. 

Case or No

Really with the glass front and back its just gotta get broken at some point.  I can see why apple moved away from it.  But it feels cozy in the hand.  the glass is heavier but warm.  metal is so "cold" in comparison  but lighter and stronger are tuff functionalities to beat. In fact, the glass makes u want to take it out of the case and feel it bareback in your hand.  It feels really nice, but danger danger when out of the case one drop and its done for!


Interface / OS

Direct updates from Google for the win.  Last 3 android phones i owned and got 1 update in 3 years total.  Swore i would never get stuck off on some android branch that there is no active development on, IE:  Had to a Nexus. 

Unlocked.  No encrypted boot loader.  This is a hackers paradise.  
Speed, ui, are all perfectly smooth no glitches cause bugs are getting fixed daily and rolled out weekly. 
iOS comparable.
I don't notice slowdowns ever (i make no effort to close apps).
Phone crashed twice in 5 months....

Battery

Battery not coming out is no biggie for me.  There are screws on the top that i'm sure one could use to put a new battery in if need be.    I'll break it before i need a new battery likely. 

I don't do alot of internetting on my phone and the batter is more than enough for me.  I keep the mobile data off and that really saves a ton of power (wifi on though).  If i turn the mobile data on I'd be lucky to get though the day.  Screen still sucks most of the juice but can't really turn that off (when using the phone). 


Storage

Lack of micro-sd sucks.  I've pretty much standardized on micro-sd between my cameras, laptop, and previous phones.  Sucks that i can't take one of my 2-gig playlists and pop it in.   Got some extra google drive storage.  Its linked to my google acct which i'm not to excited but sigh...

Camera

Camera is amazing.
Simple.  Works quickly. I hate delays. 
The best camera is the one that you have on you. 
not if it takes 10 min to get the smart phone into camera mode....
moments can be lost forever.  
Don't even need to unlock screen to get into camera mode (or flash light mode for that matter). 
(fast fast fast)

Super bright (sunny on snow = glare) with black subject  and blue skyline on a cold cold day 


Macro tests (ice crystals are a few mm in size)





Test of Vintage filter 


Nice Box!

As a Phone

Reception call quality no complaints.  Even works well with the windows down while driving!
Great build quality.  
No LTE but no worries.  
There is an LTE chip in the phone, its just not enabled right now but hackers have enabled the chip and connected to an LTE network means sometime in the future.....

Alot of people made a big stink about the lack of "4g" in the phone.   But its just bull by people who love making excuses.  The reality is it gets near LTE speeds.  Users are claiming, benchmarks are showing that the nexus 4 is  getting between 20 and 40 mbps in the real world. Which is almost exceeding the theoretical limit  to HSPA+ (does that make it HSPA++ or 3.75G as some are calling it).  Google claims to have made improvements to the standard that allow them to reach near theoretical speeds (most hspa+ phones donn't get more than 10mbps)


Conclusion

Absolutely Love My Nexus 4, perfect phone for me! 
My first new phone in forever and it's made out of glass = Really scared to break it
 =0

Android the way is was meant to be.  Updates and bug fixes on a regular basis. 
No random changes made by hardware manufactures.









1-2 of 2