Changes in v0.9¶
The biggest change is that pybedtools is now under the MIT license, following the lead of BEDTools itself.
Intervalobjects representing VCF lines now have their
startattribute correctly zero-based, as indicated in the docs and consistent with other 1-based formats. See #355.
Bugfix: Manually creating
Intervalobjects using the
otherfieldsargument now correctly converts to C++ strings. See #348.
Removed some remnants of Python 2.7 support (thanks @DavidCain)
Updates to setup.py classifiers to better reflect state of code (thanks @DavidCain)
Sorted filenames in setup.py to improve reproducibility of build (thanks @lamby)
Tests converted to run on GitHub Actions (see #339).
Changes in v0.8.2¶
2021-03-13: Minor updates
Removed scripts directory from installed version. These are still available in the GitHub repo, but were causing import issues with Python 3.9 and were not well-used in the first place.
Bugfix: unicode is better handled in gzipped files (thanks @naumenko-sa, see #320)
Bugfix: correctly ignore warnings even with capital letters (thanks @JureZmrzlikar, see #326)
Improvement: updates tests to work with bedtools v2.30
Changes in v0.8.1¶
2019-12-27: This version has minor updates and bugfixes:
Bugfix: Fixes to
pbt_plotting_example.py(thanks Steffen Möllera @smoe)
BedTool.saveas()when a BedTool already points to a compressed file correctly saves (#308)
pandas.read_table(thanks André F. Rendeiro @afrendeiro)
Improvement: overhaul testing on travis-ci
Improvement: BedTool objects support either strings or Path objects (#287, thanks @drchsl)
Improvement: MySQL host can be configured (#301, thanks André F. Rendeiro @afrendeiro)
Improvement: Proper CRAM support: #307
Improvement: Raise an exception when the
BedTool.intersecthas more than 510 files (#303)
*.hfiles now included in the distribution (thanks @blaiseli)
Improvement: Update tests to work with bedtools v2.29.2
Changes in v0.8.0¶
This version further improves testing, improves the way C++ files are included in the package, and fixes many long-standing bugs.
Using pytest framework rather than nose for testing
setup.pyto be more robust and to more clearly separate “cythonization” into .cpp files
Updated test harness for testing in independent conda environments
All issue tests go in their own test module
Included Python 3.7 tests (note that at the time of this writing, pysam is not yet available on bioconda so that dependency is pip-installed in the test) (#254)
Updated tests to reflect BEDTool 2.27.1 output (#260 <https://github.com/daler/pybedtools/issues/260>`_`#261)
contrib.classifiermodule, which has been unsupported for a while.
More informative error messages for UCSC tools if they’re missing (#227)
BedTool objects that are the result of operations that create files that are not BED/GTF/GFF/BAM can be more easily converted to pandas.DataFrame with
Added aliases to existing methods to match current BEDTools commands, e.g. the
BedTool.nucleotide_contentmethod can now also be called using
BedTool.nucwhich is consistent with the
bedtools nuccommand line name.
New wrapper for
bedtools split. The wrapper method is called
splitbedto maintain backwards compatibility because
pybedtools.BedToolobjects have long had a
splitmethod that splits intervals based on a custom function.
New wrapper for
BedTool.from_dataframehandles NaN in dataframes by replacing with
".", and is more explicit about kwargs that are passed to
Raise FileNotFoundError when on Python 3 (thanks Gosuke Shibahara, (#255)
Relocated BEDTools header and .cpp files to the
pybedtools/includedirectory, so they can more easily be linked to from external packages (#253)
Add test for (#118)
BedTool.tabix_contigswill list the sequence names indexed by tabix (#180)
BedTool.tabix_intervalswill return an empty generator if the coordinates provided are not indexed, unless
check_coordinates=Truein which case the previous behavior of raising a ValueError is triggered (#181)
Bugfix: Avoid “ResourceWarning: unclosed file” in
helpers.isBGZIP(thanks Stephen Bush)
Bugfix: Interval objects created directly no longer have their filetype set to None (#217)
BedTool.head()no longer uses an IntervalIterator (which would check to make sure lines are valid BED/GTF/GFF/BAM/SAM). Instead, it simply prints the first lines of the underlying file.
Bugfix: functions passed to
BedTool.eachno longer silently pass ValueErrors (#231)
Bugfix: Fixed IndexError in IntervalIterator if there was an empty line (#233)
Bugfix: Add additional constraint to SAM file detection to avoid incorrectly detecting a BED file as SAM (#246)
Bugfix: accessing Interval.fields after accessing Interval.attrs no longer raises ValueError (#246)
Changes in v0.7.10¶
Various bug fixes and some minor feature additions:
Support for comma-separated lists for
mapBed(thanks Chuan-Sheng Foo)
Support many calls to
tabix_intervalswithout hitting a “Too many open files” error (#190)
Clarify documentation for
genome_coveragewhen used with default parameters (#113)
Ignore stderr from samtools on older zlib versions (#209, thanks Gert Hulselmans)
Support fetching all regions from a chromosome (#201, thanks Matt Stone)
Add wrapper for
shiftBed(#200, thanks Saket Choudhary)
truncate_to_chromin Python 3 (#203, thanks Saulius Lukauskas)
When making bigWigs, use
bedSortto ensure the sort order matches that expected by UCSC tools (#178)
Fix newline handling of
Changes in v0.7.9¶
Minor bugfix release:
Changes in v0.7.8¶
Be more careful about BAM vs bgzipped files (#168)
BedTool.bgzipnow preserves the header when sorting
In Python 3, parsed BEDTools help string is decoded properly
Ensure integer number of processes in Python 3 (thanks Illa Shamovsky)
Add details on IOError messages for broken pipe error
Make converting to pandas.DataFrames easier with non-standard BED files (thanks Panos Firmpas)
Changes in v0.7.7¶
Chromsizes for dm6 and mm10 assemblies added to
Better Python 3 compatibility in the
featurefuncs.UniqueIDclass, useful for ensuring all features in a file have a unique ID in their name field.
Fix error message when a specified genome file doesn’t exist (thanks Saket Choudhary)
Changes in v0.7.6¶
pybedtools.contrib.long_range_interactionfor working with HiC-like data.
Changes in v0.7.5¶
When using tabix-indexed files,
bgzipare no longer required to be installed separately. Only
Recent BEDTools releases support multiple files for the
bedtools intersect. This version of
pybedtoolsnow supports multiple files as well. Note that it is still possible to provide a list of strings representing intervals as the
BedTool.intersect. To differentiate between a list of intervals and a list of filenames, the first item converted into an
Intervalobject; if it fails then consider the items to be filenames; otherwise assume strings representing intervals. This check only occurs if the
bargument is a list or tuple; other iterable types are always assumed to be intervals.
Changes in v0.7.4¶
Changes in v0.7.1¶
This is largely a bugfix release with the following changes:
fix for some BAM headers (thanks Gabriel Platt)
unified IntervalIterator to address some streaming issues (fixes #143)
fix bug where
__add__was not re-raising exceptions (thanks Brad Chapman and Dan Halligan)
Changes in v0.7.0¶
This release reflects a major upgrade in the underlying code in order to
support both Python 2 and Python 3 using the same code. Aside from trivial
things like converting print statements to functions and using
.next(), this required a substantial rewrite to support the way strings
are handled in Python 3 (in Cython and wrapped C++) and how relative modules
Importantly, after converting them to Python 2- and 3-compatible syntax all previous tests pass so to the end user should not notice any differences except those noted below.
Strings from Interval fields are unicode¶
For consistency between Python 2 and 3, all strings from Interval objects are now unicode. That is, in Python 2, previously we would get this:
>>> a = pybedtools.example_bedtool('a.bed') >>> a.name 'feature1'
Now, we get this:
>>> a = pybedtools.example_bedtool('a.bed') >>> a.name u'feature1'
samtools no longer a dependency¶
The dependency for samtools has been removed, which simplifies the installation
pysam is used for handling BAM files.
In order for existing tests to pass,
pysam.AlignedSegment objects are
currently converted to
pybedtools.Interval objects when iterating over a BAM
file. This will come at a performance cost if you are iterating over all reads
in a BAM file using the
In the future, iterating over a BAM file will yield
objects directly, but for now you can use the
attribute to access the underlying
Cython no longer a dependency¶
.cxx files are now shipped with the
distribution, so Cython is no longer a requirement for installation.
You will however need to have Cython installed if you’re developing pybedtools.
Remote BAM support clarification¶
pybedtools was able to support remote BAMs by loosely wrapping
samtools, but BAM files still needed to be fully downloaded to disk before
using with BEDTools. This was done automatically, but through an inefficient
Pysam does support remote BAMs, and as before, a BAM file needs to be created on disk for use with BEDTools. But now this needs to be explicitly done by the user, which should result in better performance.
Iterating over intervals¶
Previously, when iterating over a
BedTool object, different machinery would
be invoked depending on whether the BedTool was pointing to a file (a
cbedtools.IntervalFile would be invoked), to another iterator of Interval
objects, or to a stream like from the stdout of a BEDTools call
(cbedtools.IntervalIterator in both cases).
Everything is now an IntervalIterator, simplifying the path towards performance optimization.
Thanks to Saulius Lukauskas, gzip handling is now improved, and calling
BedTool.saveas() with a
.gz extension on the filename will automatically
compress the output.
In the github repo there is a
docker directory containing Dockerfiles to set
up isolated testing environments. These Dockerfiles also demonstrate how to set
up a complete environment starting from a base Ubuntu install.
All tests from v0.6.9 (which was Python 2 only) have been made Python 2/3 compatible and all previous tests pass.
If you have docker installed, from the top level directory, you can run the full tests like this:
cd docker ./full-tests.sh
This will build docker containers for Python 2 and Python 3 with all depedencies, export the parent directory to the container, and run the test suite.
You can now install the latest versions of tabix, bedtools, pysam, and
pybedtools from conda, dramatically speeding up installation time. These
mechanisms are used for automated testing as well (see the
script in the github repo).
To use these packages in your own environment(s), specify the
channel like this:
conda install -c daler pybedtools
Note that this will not install BEDTools or tabix unless you explicitly say so:
conda install -c daler pybedtools bedtools tabix
This currently only works on Linux; contributions to Mac conda recipes (see
conda dir in the github repo) would be welcomed.
Changes in v0.6.9¶
Minor bug fix release.
improved the automatic field name handling when converting an interval file to a
fixed a bug in
count_hitswhere zero-length features were being counted multiple times (thanks Brent Pedersen and Kyle Smith)
bgzip and tabix paths can now be configured separately (thanks Rob Beagrie)
fixed a bug where streaming BAM files were read fully into memory (thanks Alexey Sergushichev)
Changes in v0.6.8¶
Bugfix: Thanks to Gabriel Pratt,
pybedtools is no longer plagued by open filehandles
in the C code causing the notorious “Too many files open” error.
Changes in v0.6.7¶
Now compatible with BEDTools v2.21.0.
The one exception is that the new
bedtools intersect functionality that
-b files is not yet implemented in
BedTool.fisher()wraps the new BEDTools
fishertool. The result is an object containing parsed results.
percentileargument, useful when applying colormaps to data with a handful of extreme outliers
BedToolobject into a
pandas.DataFramewith columns named after the appropriate fields for the filetype (thanks Radhouane Aniba for the suggestion)
BedTool.head()(thanks Radhouane Aniba for the suggestion)
Add hg38 and hg38.default chromsizes
Minor bug fixes:
Ensure tuple-like args to
Temp fix for BEDTools v2.20.0 which required the
-warg to come before the
Better (i.e., UCSC Genome Browser-compliant) defaults for
Fix for BedTool.all_hits() and any_hits() which will now show hits for zero-length features intersecting with other zero-length features with the same coordinates.
Changes in v0.6.6¶
This is a compatibility release, updated for BEDTools v2.20.0.
There is one API change that affects the behavior of overloaded operators (that
- with BedTool objects) when one of the BedTool objects
represents an empty file.
a is a BedTool object representing a regular BED file but
a + b = a
b + a = b
a - b = a
b - a = a
b - b = b
a + a = a
The following changes have been made (indicated in bold), which hopefully make more logical sense:
a + b = b
b + a = b
a - b = a
b - a = b
b - b = b
a + a = a
Changes in v0.6.5¶
This is a minor bug-fix release:
Fix for BedTool.all_hits() and any_hits() which will now show hits for zero-length features with the same coordinates, like the rest of BEDTools.
Improved error-handling to avoid Python interpreter crashing in cases when a BED file on the filesystem becomes unavailable after a BedTool object has been created for it.
Changes in v0.6.4¶
Full integration with BEDTools v2.18. This includes some compatibility fixes for the new buffered output capabilities of BEDTool
intersectand wrapping the new
Overloaded operators (
-) allow empty files as input, even using BEDTools v2.18+.
Travis-CI builds now use BEDTools v2.18+ for tests.
BedTool.randomstats()(thanks Michael Reschen)
Changes in v0.6.3¶
pybedtools.parallelmodule for working with many operations in parallel. See the docs for
pybedtools.contrib.bigbed.bigbed()for converting to bigBed format, along with auto-SQL creation as needed.
pybedtools.contrib.bigbed.bigbed_to_bed(), so now bigBed -> BED and BED -> bigBed interconversions are trivial.
Support for remote BAMs by passing
remote=Truewhen creating a
BedTool.at()for subsetting a BedTool by a set of (sorted) indexes.
add_color()for applying matplotlib colormaps to BED files; see also new method
pybedtools.plotting.BinaryHeatmapclass for working with results from
BedTool.each()now also has some filter capabilities (if provided function’s return value evaluates to False, feature will be skipped)
Better detection for samtools (thanks Luca Beltrame)
Expand BEDToolsError (thanks Ryan Layer)
Creating a BedTool from a list of intervals now saves to temp file instead of treating like a consume-once iterator (#73)
Various fixes to keyword arg handling to match semantics of BEDTools.
Command line help and improved docs for the
Fix to GFF attributes (thanks Libor Mořkovský)
Fix to labels in
pybedtools.contrib.venn_maker.py(thanks Luca Pinello)
Make the naive scaling (to million mapped reads) in
BedTool.cat()to handle cases where at least one input is an empty file
Removed SciPy dependency
Every commit is built with Travis-CI for continuous integration testing of changes to source code.
Changes in v0.6.2¶
Wrapped new tools available in BEDTools 2.17:
BedTool.reldist()wrap the new
Initial implementations of building blocks for computing statistics,
pybedtools.contrib.plotting.binary_heatmap()for visualizing results from
Fixed a long-standing issue where streaming
BedToolobjects did not close their open file handles (stdout). When working with many (i.e. tens of thousands) files, this caused the operating system to hit its open file limit. This is now fixed.
BedTool.random_op(), a new mechanism for implementing operations that you would like to apply over tens of thousands of shuffled interval files. This makes it easy to extend the existing
pybedtools.contrib.bigwig.bam_to_bigwig(), a helper function to create a libary-size-scaled bigWig file from an input BAM file.
pybedtools.contrib.plotting.TrackCollectionclass, which handles plotting multiple files at once, using a provided “stylesheet” configuration to tweak colors etc.
pybedtools.contrib.plotting.ConfiguredBedToolsDemo, useful for running many graphical demos of BEDTools operations using the same “stylesheet” configuration. Run
pybedtools/contrib/plotting.pyfor a demo.
chromsizes dictionaries for common assemblies now have a
defaultattribute, which is an OrderedDict of a default set of chromosome. For example,
pybedtools.chromsizes('hg19').defaultcontains only the entries for the autosomes and X and Y.
BedTool.cat()now works better with multiprocessing
BedTool.randomstats(), which will attach the full distribution of all the randomized files to the results dictionary.
New method implementing Jaccard statistic (with pvalue using randomizations):
featurefuncs.extend_fields()helper function to pad fields with
'.', useful for manipulating features with the
Fixed a bug where BAM files, when written to disk via
BedTool.saveas(), were saved as SAM files.
Better GTF/GFF detection, and if the input had quoted attribute values, then the output will, too
various minor bug fixes and improvments as documented in the github commit logs….
Changes in v0.6.1¶
pybedtools.contrib.plotting.Trackclass allows plotting of features with matplotlib. The
matplotlib.collections.PolyCollection, making it rather fast for 1000s of features.
scripts/pbt_plotting_example.pyscript for a way of visually showing the results of BEDTools operations … great for teaching BEDTools to new users.
BedTool.liftover()method (needs a chain file from UCSC and the
BedToolcreation using tuples/lists of values – everything is converted to string before creating an
BedTool.window_maker()now handles the
pybedtools.cleanup(remove_all=True)now works correctly when using the default temp dir
Changes in v0.6¶
Overhaul in online documentation to hopefully make functionality easier to find and/or discover. See pybedtools Reference for summary tables of the different parts of
pybedtools; each entry is linked to further class/method/function-specific docs. These more detailed docs also have links to view the source code from within the HTML docs for more exploration.
pybedtools.contrib.venn_maker()function that acts as an interface to the VennDiagram R package – just give it some BED files and it’ll do the rest.
Debug mode –
pybedtools.debug_mode()– for verbose logging messages.
Fixed an open file leak (OSError: too many open files) that occured when opening thousands of streaming bed files in a single session.
Initial support for tabix files. Useful for extracting features from a single region when you don’t need a full intersection.
pybedtools.contribmodule (in the spirit of Django’s
contrib) where higher-level functionality will be built.
pybedtools.contrib.Classifierclass for identifying the classes of intervals. Useful for making pie charts of intronic/exonic/intergenic etc classes of peaks. Note that this is somewhat redundant with the new
mapBedprogram in BEDTools.
pybedtools.contrib.IntersectionMatrixclass for handling pairwise intersections of a large number of interval files – including a local sqlite3 database to avoid re-computing already up-to-date results.
Intervalobjects are now hashable (it’s just a hash of the string representation) so that you can use them as dictionary keys.
BedTool.split()method, which accepts a function returning an iterable of
Intervalobjects. The function is applied to each interval. Useful for, say, splitting each gene into TSS, TTS, upstream and downstream features.
BedTool.truncate_to_chrom()method, which truncates features to the chromosome sizes of the provided genome. Useful for when you try uploading a MACS-generated track to the UCSC genome browser, but it complains because peak boundaries have been extended outside chromosome boundaries … this method fixes the problem.
BedToolobjects now have full functionality of
IntervalFileobjects – that is, they have the methods
BedTool.count_hits()for doing single-interval tests. Sometimes this will be faster than using the tabix support, sometimes it won’t – it’s best to try both, depending on your data.
String representations of
Intervalobjects now have a newline at the end, just like a raw lines from a BED/GFF/VCF file. Previously, this was inconsistent and sometimes led to extra blank lines in “streaming”
BedToolinstances … which in turn led to problems with BEDTools programs using the chromsweep algorithm.
Concatentate multiple files with one call to
BedTool.cat()(thanks Jake Biesinger)
- Wrapped previous BEDTools programs:
- Wrapped new BEDTools programs:
Made venn_gchart and venn_mpl tests more stable
Automatic documenting of which args are passed implicitly for BedTool method calls
More robust mechanisms for specifying custom paths for BEDTools installation as well as optional tabix, samtools, and R installations. This makes it easier to explicitly specify which versions of the tools to use.
Improvements to GFF attributes: handle unescaped “=” (from sim4db GFFs) and make Attribute class properly dict-like (thanks Libor Mořkovský)
Changes in v0.5.5¶
additional_argskwarg to pass arguments verbatim to the underlying BEDTools programs. This is necessary for arguments like
5=Trueis not a valid Python expression. For example, you can use:
import pybedtools a = pybedtools.example_bedtool('a.bed') a.genome_coverage(bg=True, strand='+', genome='hg19', additional_args='-5')
Brent Pedersen added support for just 2 BED files in the Venn diagram scripts
BedTool.all_hits()uses the underlying BEDTools C++ API to get all hits in a file for a particular Interval:
a = pybedtools.example_bedtool('a.bed') interval = Interval('chr1', 1, 5000) a.all_hits(interval)
New semantics for comparisons of Interval objects. Visual documentation of this coming soon.
More tests for latest BEDTools code
Interval instances are now pickleable; they can now be used across processes for parallel code.
Changes in v0.5¶
support for running random intersections in parallel. See
BedTool.randomintersection()(thanks, Jake Biesinger)
Interval.__copy__()for compatibility with
IntervalFileclass, used for Aaron Quinlan’s new chromsweep algorithm (https://github.com/arq5x/chrom_sweep) (thanks, Aaron)
support and tests for new BEDTools programs
output="out.bed"kwarg for all wrapped methods for explicitly specifying where to save output – no more moving tempfiles
- docs improvements:
direct comparison with a shell script to illustrate benefit of
pybedtools; see Shell script comparison
more installation details
0- and 1-based coordinates discussed early on (the 3 brief examples page, Three brief examples)
development history and open collaboration model (see pybedtools development model)