I was recently asked by a client to create a large number of “proof of concept” visualizations that illustrated the power of R for compiling and analyzing disparate datasets. The client was specifically interested in automated analyses of global data. A little research led me to the WDI package.

The WDI package is a tool to “search, extract and format data from the World Bank’s World Development Indicators” (WDI help). In essence, it is an R-based wrapper for the World Bank Economic Indicators Data API. When used in combination with the information on the World Bank data portal it provides easy access to thousands of global datapoints.

Here is an example use case that illustrates how simple and easy it is to use, especially with a little help from the countrycode and ggplot2 packages:

library(WDI)
library(ggplot2)
library(countrycode)

# Use the WDIsearch function to get a list of fertility rate indicators
indicatorMetaData <- WDIsearch("Fertility rate", field="name", short=FALSE)

# Define a list of countries for which to pull data
countries <- c("United States", "Britain", "Sweden", "Germany")

# Convert the country names to iso2c format used in the World Bank data
iso2cNames <- countrycode(countries, "country.name", "iso2c")

# Pull data for each countries for the first two fertility rate indicators, for the years 2001 to 2011
wdiData <- WDI(iso2cNames, indicatorMetaData[1:2,1], start=2001, end=2011)

# Pull out indicator names
indicatorNames <- indicatorMetaData[1:2, 1]

# Create trend charts for the first two indicators
for (indicatorName in indicatorNames) { 
  pl <- ggplot(wdiData, aes(x=year, y=wdiData[,indicatorName], group=country, color=country))+
    geom_line(size=1)+
    scale_x_continuous(name="Year", breaks=c(unique(wdiData[,"year"])))+
    scale_y_continuous(name=indicatorName)+
    scale_linetype_discrete(name="Country")+
    theme(legend.title=element_blank())+
    ggtitle(paste(indicatorMetaData[indicatorMetaData[,1]==indicatorName, "name"], "\n"))
  ggsave(paste(indicatorName, ".jpg", sep=""), pl)
}

This code can be adapted to quickly pull and visualize many pieces of data. Even if you don’t have an analytic need for the WDI data, the ease of access and depth of information available via the WDI package make them perfect for creating toy examples for classes, presentations or blogs, or conveying the power and depth of available R packages.

The post Global Indicator Analyses with R appeared first on ProgrammingR.

QuantMarketplace.com connects those in need of quantitative analysis or statistical consulting services for short to medium length projects with quantitative service providers. If you or your organization needs statistical or quantitative research expertise you can post a project description and receive bids from service providers. Service providers can register and post their profile so it as available for searching by potential clients.

Website: www.quantmarketplace.com

The post QuantMarketplace.com appeared first on ProgrammingR.

The position you are applying for is connected with a large bank initiative called Enterprise Target State Fraud Detection and Monitoring. Treasury Services is the largest wholesale bank in the world; and as a result, you have the rare opportunity to analyze the largest and most important transactional data set in existence anywhere. We use state of the art statistical methods to analyze this data, identify potential fraud, and raise alerts. Big Data meets IT Security!

1. Work in a Hadoop-centric environment capable of handling massive data volume accessed through Map/Reduce parallel software in the “R” programming language
2. Create statistical technology that identifies risky transactions which may be fraudulent
3. Encourages (expects!) you to contribute both technology and business ideas on how  you should fundamentally use its Big Data environment to isolate fraud.
4. Encourages (demands!) that you think beyond the project into a wider roll-out into a mass array of applications.
5. Work in the context of the industry leaders in financial IT security. This team already used statistical methods to break encryption algorithms and analyze biometric authentication. Next, we will address fraud.
Once this multi-year project is complete you will continue operating in the same environment with ownership over the systems you created, to modify, extend, and continue creating innovative capabilities. Your core application provides technology support for our flagship applications as well as others. You will expand upon your “R” expertise incorporating statistical methods and data analysis technology to identify new fraud patterns and quickly react when our adversaries strike.

As an developer, you will provide high quality technology solutions that address business needs by developing applications within mature technology environments. You will utilize programming methodologies, create “R” coding standards, procedures and techniques, while contributing to the technical code documentation.

You will participate in project planning sessions with project managers, business analysts and team members to analyze business requirements and outline the proposed IT solution. You will participate in design reviews and provide input to the design recommendations; incorporate security requirements into design; and provide input to information/data flow, and understand and comply with Project Life Cycle Methodology in all planning steps. You will also adhere to IT Control Policies throughout design, development and testing and incorporate Corporate Architectural Standards into application design specifications. Additionally, you will document the detailed application specifications, translate technical requirements into programmed application modules and develop/enhance software application modules. You will participate in code reviews and ensure that all solutions are aligned to pre-defined architectural specifications; identify/troubleshoot application code-related issues; and review and provide feedback to the final user documentation. You will research and resolve coding defects and document root cause.

Qualifications
• A bachelor’s degree in Computer Science or equivalent experience
• 5+ years experience with full development lifecycle from inception through implementation
• Development experience with “R”
• 3+ years experience working in both Unix or Linux
• Experience with statistic techniques and data analysis
• Excellent analytical, communication, organizational and problem-solving skills coupled with a strong work ethic
• Strong planning and organizational skills
• Solid design, coding, testing and debugging skills
• Ability to translate business requirements into functional requirements documentation
• Demonstrated ability to acquire technical knowledge and skills quickly
• Ability to work in a team environment under aggressive timelines
Familiarity with Enterprise Systems applications preferred

To apply, please contact: Yenejah Jemmott at yjemmott@ttiofusa.com

The post Statistical Programmer appeared first on ProgrammingR.

Marketing professor with short-term project (5+ hours) to program in R a Bayesian statistical model.  Must have experience with Bayesian Stats.  Need output in an excel file.  Experience with Netlogo preferred but not necessary.

Please contact Rosanna Garcia at r.garcia@neu.edu.

The post Bayesian statistical programmer needed for 5 month contract project appeared first on ProgrammingR.

Chanalytics Research is seeking to fill a 6-month contract R programmer position. This contract may be renewed or the position made permanent at the end of the contract.

Job duties:

- Write code in support of R-based statistical analyses and applications
- Locate and acquire new clients
- Write proposals
- Research new R packages and statistical methods
- Present deliverables to clients (via Web conferences and in-person meetings)
- Other duties associated with automating and conducting statistical research

Job details:

- This is a remote/tele-work position; some travel may be required
- Salary will be commensurate will experience and expertise

The ideal candidate:

The ideal candidate will be self-motivated and able to work independently. He or she will have at least 2 years experience developing R code for statistical analyses or applications. This position will interface with clients and will involve generating business, so the candidate must be comfortable assisting with sales contacts. The preferred candidate will also have experience in one of the following topical areas:

- Quantitative finance
- Marketing research
- Health research
- Bayesian analysis
- Big data analysis
- The Semantic Web
- Web design

About Chanalytics Research:

Chanalytics Research is a Durham, NC based research group that is internationally known as a provider of quality analytic and statistical programming solutions. Our clients come from a variety of industries, including marketing, finance, government and academics. We pride ourselves on being agile, adaptive and diligent.

To apply, please submit resumes with cover letters to consulting@chanalyticsresearch.com.

The post Statistical Programmer/Analyst at Chanalytics Research appeared first on ProgrammingR.

ProgrammingR recently featured a review of DeVries and Meys’ R for Dummies. We were very impressed with the book and felt it would be a good addition to most R programmers’ libraries. To that end, we’re giving away a fresh new copy to a random Twitter user who retweets our newest article covering SPARQL, the Semantic Web and R. To enter, you must retweet the exact tweet below (so that we can track your entry):

https://twitter.com/ProgrammingR/statuses/294145982311063552

We’ll select a random retweeter on 2/1/13. You must have retweeted the link above by 12PM EST on 1/31/13 and this type of contest must be legal in your area to be eligible.

The post R for Dummies Twitter Contest appeared first on ProgrammingR.

In this article we’ll get up and running on the Semantic Web in less than 5 minutes using SPARQL with R. We’ll begin with a brief introduction to the Semantic Web then cover some simple steps for downloading and analyzing government data via a SPARQL query with the SPARQL R package.

What is the Semantic Web?

To newcomers, the Semantic Web can sound mysterious and ominous. By most accounts, it’s the wave of the future, but it’s hard to pin down exactly what it is. This is in part because the Semantic Web has been evolving for some time but is just now beginning to take a  recognizable shape (DuCharme 2011).  Detailed definitions of the Semantic Web abound, but simply put, it is an attempt to structure the unstructured data on the Web and to formalize the standards that make that structure possible. In other words, it’s an attempt to create a data definition for the Web.

The primary method for accessing data made available on the Semantic Web is via SPARQL queries to a data provider’s endpoint. Endpoints are portals to data that a provider has made available for querying. This data is often published in RDF format. If you’re familiar with using SQL to query a database, then the idea of using a query language (SPARQL) to access a structured database or data repository (the endpoint) should be common ground. The practice is also similar to webscraping an XML document with a known structure.

The Semantic Web and R

A primary goal of the Semantic Web movement is to enable machines and applications to interface using standardized data definitions. As the Semantic Web grows, it will be ripe for mining with statistical programs that already lend themselves to automation – such as R. To that end, let’s dive into a simple query that will illustrate 75% of what you’ll be doing on the Semantic Web.

Accessing Data.gov datasets with SPARQL

We’ll use data at the Data.gov endpoint for this example. Data.gov has a wide array of public data available, making this example generalizable to many other datasets. One of the key challenges of querying a Semantic Web resource is knowing what data is accessible. Sometimes the best way to find this out is to run a simple query with no filters that returns only a few results or to directly view the RDF. Fortunately, information on the data available via Data.gov has been cataloged on a wiki hosted by Rensselaer. We’ll use Dataset 1187 for this example. It’s simple and has interesting data – the total number of wildfires and acres burned per year, 1960-2008.

R code

Data.gov provides a Virtuoso endpoint here, which we could use to submit manual queries. But we want to automate this process, so we’ll use Willem Robert van Hage and Tomi Kauppinen’s SPARQL package to access the endpoint.

library(SPARQL) # SPARQL querying package
library(ggplot2)

# Step 1 - Set up preliminaries and define query
# Define the data.gov endpoint
endpoint <- "http://services.data.gov/sparql"

# create query statement
query <-
"PREFIX  dgp1187: <http://data-gov.tw.rpi.edu/vocab/p/1187/>
SELECT ?ye ?fi ?ac
WHERE {
?s dgp1187:year ?ye .
?s dgp1187:fires ?fi .
?s dgp1187:acres ?ac .
}"

# Step 2 - Use SPARQL package to submit query and save results to a data frame
qd <- SPARQL(endpoint,query)
df <- qd$results

# Step 3 - Prep for graphing

# Numbers are usually returned as characters, so convert to numeric and create a
# variable for "average acres burned per fire"
str(df)
df <- as.data.frame(apply(df, 2, as.numeric))
str(df)

df$avgperfire <- df$ac/df$fi

# Step 4 - Plot some data
ggplot(df, aes(x=ye, y=avgperfire, group=1)) +
geom_point() +
stat_smooth() +
scale_x_continuous(breaks=seq(1960, 2008, 5)) +
xlab("Year") +
ylab("Average acres burned per fire")

ggplot(df, aes(x=ye, y=fi, group=1)) +
geom_point() +
stat_smooth() +
scale_x_continuous(breaks=seq(1960, 2008, 5)) +
xlab("Year") +
ylab("Number of fires")

ggplot(df, aes(x=ye, y=ac, group=1)) +
geom_point() +
stat_smooth() +
scale_x_continuous(breaks=seq(1960, 2008, 5)) +
xlab("Year") +
ylab("Acres burned")

# In less than 5 mins we have written code to download just
# the data we need and have an interesting result to explore!

If you’re familiar with R, the only foreign code is that in Steps 1 and 2. There we define the location of the endpoint and write the SPARQL query. If you have experience with SQL, you’ll notice that SPARQL syntax is very similar. The most notable exception is that Semantic Web data is structured around chunks of data that contain three pieces of information. In Semantic Web terminology these three pieces of information as referred to as subjects, predicates and objects and together they are called “triples”. A SPARQL query returns pieces of these chunks of data; exactly which pieces are returned depends on the query. We have defined a prefix which establishes that everywhere we say “dgp1187:” we want it to know that we mean “<http://data-gov.tw.rpi.edu/vocab/p/1187/>”. This saves us from having to retype long URIs every time we need to reference them. URIs are used extensively on the Semantic Web, so this is a very helpful feature.

In English, our query says, “Give me the values for the attributes “fires”, “acres” and “year” wherever they are defined, and assign them to variables named “fi”, “ac” and “ye” respectively. Also fetch the location of the value in the data as a variable called ?s and merge the other data values by that variable.”

The last bit of the query is the closest we get to SPARQL magic. Because the variable ?s is present in each clause of the query, our fire, acres burned, and year data will be merged by that variable. This is somewhat analogous to saying, “Once you’ve found a row of data that contains information on the number of fires, also return the acres burned and year data from that row and list them on the same row in the new dataset.” (If the logic behind the query isn’t yet clear, or you’re ready to see how you can make this query more advanced or parsimonious, the text and web pages in the “References” and “Additional Information” sections at the end of this article are good resources for learning SPARQL.)

Once defined, we pass this query and endpoint information to the SPARQL function which returns an object with two values – a data frame of the new data and a list of namespaces. At this point, we’re concerned with the data frame, so we pull it out in the last line of Step 2.

From that point on we treat the data just like any other dataset. Running some quick graphs, we see that although the number of fires per year have decreased, the number of acres burned per fire and the number of acres burned per year increased considerably between 1975 and 2008. (Despite being born in 1975, I disavow any responsibility for this trend.)

Summary

In a few short minutes, using R and SPARQL, we wrote code to pull and do initial analyses on an interesting government dataset. Hopefully, the power of R for mining the Semantic Web is evident from this simple example. As more data becomes available in RDF format, automated solutions for mining and analyzing the Semantic Web will become more and more useful.

Reference:

DuCharme, Bob (2011-07-14). Learning SPARQL . OReilly Media.

Additional Information:

Basic tutorial on querying Data.gov

More detailed tutorial on querying Data.gov

The post SPARQL with R in less than 5 minutes appeared first on ProgrammingR.

OpenAnalytics is an international company providing data analysis
services and products across geographies and industrial sectors.

We are currently looking to expand our bioinformatics team with a
candidate who fits the following profile:

Bioinformatician: MSc / PhD in Computational Biology, Computer Science or Bioinformatics

A detailed description can be found at

http://www.openanalytics.eu/jobs

Strong R and BioConductor skills are an important asset for the
position.

The post Bioinformatician at OpenAnalytics appeared first on ProgrammingR.

The for Dummies series has been around since 1991. (A bit of trivia, DOS for Dummies was the first title.) I’ve owned a few books in the series and have been adequately impressed with most of them, but when I learned there was an R for Dummies I was immediately skeptical. Possibly I was skeptical because R has a steep learning curve and many idiosyncrasies, so the idea of an R for Dummies text seemed oxymoronic – it’s difficult to imagine a (successfully) dumbed-down version of an introductory R text. But if you’re familiar with the for Dummies series, you already know that the moniker is just for marketing. In reality, these books usually do a good job of distilling a topic down to the important components a new user needs to know. This edition is no exception.

The core topic areas that R for Dummies covers should come as no surprise: A basic overview of R and its capabilities, importing data into R, writing and debugging functions, summarizing data and graphing. In addition, there are sections covering potentially frustrating tasks for beginners such as working with dates and multidimensional arrays.

I am a big fan of periodically reviewing the fundamentals. I pick up introductory texts every once and a while to make sure I haven’t forgotten anything important and R for Dummies is a nice edition to my library for that reason.

One thing that sets this book apart from other currently available introductory R texts is that it covers a couple of recent and important developments in R coding – namely the RStudio development environment and ggplot2 graphics.

If you’re not familiar with the for Dummies series, it is important to note that they are written in a specific informal style, which is in stark contrast to most R texts. (For reference, the style is more similar to The R Inferno than the ggplot2 manual.) You can get a sense of this style by browsing a few pages on Amazon to see if you find it helpful or distracting. On the other hand, R for Dummies has a more polished feel than many R texts I’ve read. I didn’t encounter any of the frustrating and distracting editing errors that are common in some R texts.

R for Dummies is primarily focused on R as a programming language, so for the most part, statistical analyses are presented only as a means of illustrating programming techniques. Given its focus on programming and fundamentals, this book is highly recommended for someone with little to no experience in R who wants to learn R programming. Intermediate to advanced R programmers like myself who want a current review of the fundamentals, might also find it useful.

I might also recommend R for Dummies to experienced users of other programming languages who are new to R. The discussion of basic programming concepts, such as control flow, is minimal and focused primarily on details specific to R. It is not recommended for those looking to learn statistics in conjunction with R.

The current price of $20 USD puts it in the middle price range for texts of its kind. It is available as a paperback or Kindle text.

The post R for Dummies – De Vries and Meys (2012) appeared first on ProgrammingR.

Just in time for the holidays, ProgrammingR is giving you the opportunity to win a gift for yourself or someone who loves R. And in the spirit of the holidays, you’ll also be giving back to the R community.

In the last article I presented two helper functions I use often. I also teased an upcoming, related event. This is that event.

The premise is simple. Post your favorite (original) R helper function(s). We’ll select 3-5 of the top candidates and put those to a popular vote. The author of the function with the most votes at the end of the contest will receive their choice of one of the three books below.

The contest begins now. Post your submissions to this forum or use the comments below to point us to a post on your site written specifically for this contest. You can enter as many of your helper functions as you like between now and midnight 12/14/12 Eastern Standard Time. We’ll choose the finalists and allow approximately a week of voting for them starting soon after 12/14. The author of the function with the most votes at the end of the popular voting will be the winner.

I look forward to seeing what the ProgrammingR community has to offer!

Some Important Guidelines:
By participating, you agree to be bound by the decisions of the ProgrammingR staff in all contest-related aspects, but especially in regard to who the finalists are and the winner is. We will do our best to run a fair and impartial contest, but we can’t guarantee the originality of every submission, for example. You also certify that you have the right to publicly post any code you submit and that we can repost it on the site. If you point us to a submission on your site and your code is selected as a finalist, we’ll need to verify that it actually is your site.

The post Helper Function Contest appeared first on ProgrammingR.