Hyndsight on Rob J Hyndman
https://robjhyndman.com/hyndsight/
Recent content in Hyndsight on Rob J HyndmanHugo -- gohugo.ioWed, 12 Jul 2017 00:00:00 +0000IJF Best Paper Award 2017
http://feedproxy.google.com/~r/Hyndsight/~3/4D94F_tTRNk/
Wed, 12 Jul 2017 00:00:00 +0000https://robjhyndman.com/hyndsight/ijf-best-paper-award-2017/At the recent International Symposium on Forecasting, I announced the awards for the best paper published in the International Journal of Forecasting in the period 2014–2015.
We make an award every two years to the best paper(s) published in the journal. There is always about 18 months delay after the publication period to allow time for reflection, citations, etc. The selected papers are selected by vote of the editorial board. The best paper wins an engraved bronze plaque and US$1000.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/4D94F_tTRNk" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/ijf-best-paper-award-2017/IIF Tao Hong Award 2016
http://feedproxy.google.com/~r/Hyndsight/~3/PbJlCaQkrWY/
Sun, 09 Jul 2017 00:00:00 +0000https://robjhyndman.com/hyndsight/iif-hong-award-2016/A generous donation from Professor Tao Hong has funded this new award for papers on energy forecasting published in the International Journal of Forecasting. The award for 2016 is for papers published within 2013–2014. Next year we will award a paper published in 2015–2016, and we will make the award every two years after that.
The award decision was made by a committee consisting of Professors Pierre Pinson, James Mitchell and Rob J Hyndman.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/PbJlCaQkrWY" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/iif-hong-award-2016/IIF Fellow Ralph Snyder
http://feedproxy.google.com/~r/Hyndsight/~3/luprf01khOw/
Fri, 07 Jul 2017 00:00:00 +0000https://robjhyndman.com/hyndsight/ralph-snyder/<p><img src="https://robjhyndman.com/img/Ralph.jpg" width=200></p>
<p>At the <em>International Symposium on Forecasting</em> last week, my friend and colleague <a href="https://scholar.google.com/citations?user=mlwKz6cAAAAJ&hl=en">Ralph Snyder</a> was made a Fellow of the International Institute of Forecasters.
</p><img src="http://feeds.feedburner.com/~r/Hyndsight/~4/luprf01khOw" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/ralph-snyder/Hex stickers for the forecast package
http://feedproxy.google.com/~r/Hyndsight/~3/EDYRM98NID4/
Mon, 26 Jun 2017 00:00:00 +0000https://robjhyndman.com/hyndsight/forecast-stickers/I’ve caved in to the hex sticker craze, and produced some hex stickers for the forecast package for R. If you attend a workshop I teach, I’ll give you one. Otherwise you can order (in bulk) from hexi.pics.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/EDYRM98NID4" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/forecast-stickers/Why I'm not celebrating the 2016 impact factors
http://feedproxy.google.com/~r/Hyndsight/~3/FXopg5Pr5aE/
Wed, 21 Jun 2017 00:00:00 +0000https://robjhyndman.com/hyndsight/2016-impact-factors/Once every year, the journal citation reports are released including journal impact factors. This year, the International Journal of Forecasting 2-year impact factor has increased to 2.642 which is the highest it has been in the journal’s history, and puts the journal higher than such notable titles as Journal of the American Statistical Association and just below Management Science.
The 2-year impact factor is the average number of citations for articles published in the previous 2 years.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/FXopg5Pr5aE" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/2016-impact-factors/My new DataCamp course: Forecasting Using R
http://feedproxy.google.com/~r/Hyndsight/~3/dakOdYn2Olw/
Thu, 01 Jun 2017 00:00:00 +0000https://robjhyndman.com/hyndsight/datacamp/For the past few months I’ve been working on a new DataCamp course teaching Forecasting using R. I’m delighted that it is now available for anyone to do.
Course blurb Forecasting involves making predictions about the future. It is required in many situations such as deciding whether to build another power generation plant in the next ten years requires forecasts of future demand; scheduling staff in a call center next week requires forecasts of call volumes; stocking an inventory requires forecasts of stock requirements.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/dakOdYn2Olw" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/datacamp/Prediction intervals for NNETAR models
http://feedproxy.google.com/~r/Hyndsight/~3/t_moNzi12uc/
Thu, 25 May 2017 00:00:00 +0000https://robjhyndman.com/hyndsight/nnetar-prediction-intervals/The nnetar function in the forecast package for R fits a neural network model to a time series with lagged values of the time series as inputs (and possibly some other exogenous inputs). So it is a nonlinear autogressive model, and it is not possible to analytically derive prediction intervals. Therefore we use simulation.
Suppose we fit a NNETAR model to the famous Canadian lynx data:
library(forecast) (fit <- nnetar(lynx, lambda=0.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/t_moNzi12uc" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/nnetar-prediction-intervals/ISI Karl Pearson Prize for 2017
http://feedproxy.google.com/~r/Hyndsight/~3/vnr7h7v8fQQ/
Tue, 23 May 2017 00:00:00 +0000https://robjhyndman.com/hyndsight/kpprize2017/Recently I was privileged to sit on the committee that selects the winner of the Karl Pearson Prize. KP was, of course, an early mathematical statistician, famous for many commonly-used statistical methods and tools including histograms, the correlation coefficient, the method of moments, p-values, the chi-squared test and principal components analysis. He is also infamous for his highly racist views, support for eugenics, anti-semitism and for refusing a knighthood.
All that aside, the job of the committee was to select an English-language article or book published in the last 30 years that has made a stand-alone research contribution, and which has had major influence on one or more of statistical theory, statistical methodology, statistical practice and application.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/vnr7h7v8fQQ" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/kpprize2017/Academic vacancies at Monash
http://feedproxy.google.com/~r/Hyndsight/~3/az3JJXXYX3Q/
Thu, 18 May 2017 00:00:00 +0000https://robjhyndman.com/hyndsight/jobs-may-2017/We are hiring again, at all academic levels. While all fields of specialisation within econometrics and statistics are open, expertise in any of Business Analytics, Data Science, Applied Econometrics and Actuarial Science is particularly encouraged.
All positions are full-time and continuing (i.e., tenured after a probation period).
Applications close at 11:55pm AEST, Friday 30 June 2017.
Please direct initial questions to Liam Mahon.
Lecturer Suitable for someone just completing a PhD or a post-doc.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/az3JJXXYX3Q" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/jobs-may-2017/Handgun acquisitions in California after two mass shootings
http://feedproxy.google.com/~r/Hyndsight/~3/FBTuNyP4KO4/
Thu, 04 May 2017 00:00:00 +0000https://robjhyndman.com/hyndsight/handguns/<p>My new paper <a href="https://robjhyndman.com/publications/handguns/">(Studdert et al, 2017)</a> on “Handgun acquisitions in California after two mass shootings” has been attracting some press. Here are some of the news items I’ve found:</p><img src="http://feeds.feedburner.com/~r/Hyndsight/~4/FBTuNyP4KO4" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/handguns/Monthly seasonality
http://feedproxy.google.com/~r/Hyndsight/~3/I-j1y3ZdzvA/
Wed, 03 May 2017 00:00:00 +0000https://robjhyndman.com/hyndsight/monthly-seasonality/I regularly get asked why I don’t consider monthly seasonality in my models for daily or sub-daily time series. For example, this recent comment on my post on seasonal periods, or this comment on my post on daily data. The fact is, I’ve never seen a time series with monthly seasonality, although that does not mean it does not exist.
Monthly seasonality is defined as a regular pattern that recurs every month in data that is observed more frequently than monthly.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/I-j1y3ZdzvA" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/monthly-seasonality/Converting to blogdown
http://feedproxy.google.com/~r/Hyndsight/~3/41Bo1f0RICA/
Sun, 30 Apr 2017 00:00:00 +0000https://robjhyndman.com/hyndsight/blogdown/This website has gone through several major updates over the years. It began in 1993 as some handcrafted html files, transitioned to Joomla and later to Wordpress. Then it slowly grew into a collection of ten connected Wordpress installations that became increasingly difficult to maintain, and rather slow.
So I’ve now converted the entire site to Blogdown/Hugo. Nearly 700 pages of wordpress content have been translated to markdown. I decided to drop a few parts of the site, notably the pages for my 1998 forecasting textbook.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/41Bo1f0RICA" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/blogdown/Software for honours students
http://feedproxy.google.com/~r/Hyndsight/~3/SQelglOrgG8/
Tue, 04 Apr 2017 23:00:58 +0000https://robjhyndman.com/hyndsight/software-for-honours-students/I spoke to our new crop of honours students this morning. Here are my slides, example files and links.
Managing References Mendeley Zotero Paperpile Data analysis and computation Download R Download Rstudio Online R tutorial R packages for time series R packages for econometrics R packages for finance Writing your thesis LaTeX Windows: Download MikTeX Mac OSX: Download MacTeX Linux: Check your usual software source for TeXLive; otherwise install TeX Live directly.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/SQelglOrgG8" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/software-for-honours-students/Monash Rmarkdown templates on github
http://feedproxy.google.com/~r/Hyndsight/~3/pkR49pbggqU/
Thu, 30 Mar 2017 21:53:14 +0000https://robjhyndman.com/hyndsight/monash-rmarkdown-templates-on-github/My Rmarkdown templates for staff and students in my department are now available as github repositories.
MonashThesis: For PhD theses. MonashBeamer: For beamer slides. MonashWP: For working papers. Feel free to fork and adapt to your own purposes.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/pkR49pbggqU" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/monash-rmarkdown-templates-on-github/Academic phishing
http://feedproxy.google.com/~r/Hyndsight/~3/N5MS8z2kego/
Sat, 18 Mar 2017 02:25:27 +0000https://robjhyndman.com/hyndsight/academic-phishing/Invitations to write for bogus journals and speak at bogus conferences keep rolling in. Here is one I received today.
Dear Dr. Rob J. Hyndman, It is our great pleasure to welcome you to join in Part 2: Knowledge Economy Symposium of GCKE-2017, which will be held in Qingdao, China during September 19-21, 2017. And we cordially invite you to propose a Speech on your recent research of Corrigendum to: “Hierarchical forecasts for Australian domestic tourism” [International Journal of Forecasting 25 (2009) 146–166]… .<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/N5MS8z2kego" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/academic-phishing/Follow-up forecasting forum in Eindhoven
http://feedproxy.google.com/~r/Hyndsight/~3/suiDA6i001I/
Tue, 07 Mar 2017 22:35:08 +0000https://robjhyndman.com/hyndsight/eindhoven2/Last October I gave a 3-day masterclass on “Forecasting with R” in Eindhoven, Netherlands. There is a follow-up event planned for Tuesday 18 April 2017. It is particularly designed for people who attended the 3-day class, but if anyone else wants to attend they would be welcome.
Please register here if you want to attend.The preliminary schedule is as follows.
10.00 -- 11.00 New developments in forecasting using R forecast v8.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/suiDA6i001I" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/eindhoven2/WOMBAT MeDaScIn 2017
http://feedproxy.google.com/~r/Hyndsight/~3/tMtdHGQNGiw/
Tue, 07 Mar 2017 06:37:56 +0000https://robjhyndman.com/hyndsight/wombat-medascin-2017/Last year we had WOMBAT (Workshop Organized by the Monash Business Analytics Team) at the zoo, and MeDaScIn (Melbourne Data Science Initiative) in the city.
This year we are combining forces to hold WOMBAT MeDaScIn 2017.
There will be four days of tutorials (Monday 29 May to Thursday 1 June), and the main conference on Friday 2 June. We have an impressive range of local and international presenters including Yihui Xie (author of Rmarkdown, Knitr, Bookdown, Blogdown and more), Di Cook (data visualization guru), Stephanie Kovalchik (Data Scientist at Tennis Australia), Amy Shi-Nash (Head of Data Science at Commonwealth Bank of Australia), Graham Williams (Director of Data Science at Microsoft) and many more.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/tMtdHGQNGiw" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/wombat-medascin-2017/IJF Best Paper Award 2014-2015
http://feedproxy.google.com/~r/Hyndsight/~3/S-868rhmH_Y/
Wed, 01 Mar 2017 03:54:34 +0000https://robjhyndman.com/hyndsight/ijf-nominations1415/Every two years we award a prize for the best paper published in the International Journal of Forecasting. It is now time to identify the best paper published in the IJF during 2014 and 2015. There is always about 18 months delay after the publication period to allow time for reflection, citations, etc. The prize is US$1000 plus an engraved plaque. I will present the prize at the ISF in Cairns in late June.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/S-868rhmH_Y" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/ijf-nominations1415/forecast 8.0
http://feedproxy.google.com/~r/Hyndsight/~3/0HVfYSG5lJ0/
Tue, 28 Feb 2017 14:00:53 +0000https://robjhyndman.com/hyndsight/forecast8/In what is now a roughly annual event, the forecast package has been updated on CRAN with a new version, this time 8.0.
A few of the more important new features are described below.
Check residuals A common task when building forecasting models is to check that the residuals satisfy some assumptions (that they are uncorrelated, normally distributed, etc.). The new function checkresiduals makes this very easy: it produces a time plot, an ACF, a histogram with super-imposed normal curve, and does a Ljung-Box test on the residuals with appropriate number of lags and degrees of freedom.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/0HVfYSG5lJ0" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/forecast8/Invited sessions at ISF2017
http://feedproxy.google.com/~r/Hyndsight/~3/-9PMNF_Ji-A/
Mon, 27 Feb 2017 22:29:29 +0000https://robjhyndman.com/hyndsight/invited-sessions-at-isf2017/We are still looking for a few more invited sessions for the International Symposium on Forecasting, to be held in Cairns, Australia, 25-28 June 2017.An invited session consists of 3 or 4 talks around a specific, forecasting theme. You are allowed to be one of the speakers in a session you organize (although it is not necessary). Therefore, if you know what you are planning to speak about, all you need to do is find 2 or 3 other speakers who will speak on a related topic, and invite them.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/-9PMNF_Ji-A" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/invited-sessions-at-isf2017/Forecasters: bring your family to Cairns
http://feedproxy.google.com/~r/Hyndsight/~3/vhf-REELIPU/
Wed, 15 Feb 2017 10:06:20 +0000https://robjhyndman.com/hyndsight/isf-social/Courtesy of Tourism Tropical North Queensland We know Australia is a long way to come for many forecasters, so we are making it easy for you to bring your families along to the International Symposium on Forecasting and have a vacation at the same time.
During the International Symposium on Forecasting, there will be a social program organized for family and friends. Monday, 26 June Official Partners Tour: Kuranda, Train & Skyrail.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/vhf-REELIPU" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/isf-social/The Australian Macro Database
http://feedproxy.google.com/~r/Hyndsight/~3/oVrC7B3n_Ms/
Wed, 15 Feb 2017 00:40:19 +0000https://robjhyndman.com/hyndsight/ausmacrodata/AusMacroData is a new website that encourages and facilitates the use of quantitative, publicly available Australian macroeconomic data. The Australian Macro Database hosted at ausmacrodata.org provides a user-friendly front end for searching among over 40000 economic variables and is loosely based on similar international sites such as the Federal Reserve Economic Database (FRED). In total, data on 40,304 variables are available for download from AusMacroData. The majority of variables are sourced from the Australian Bureau of Statistics (ABS) and include data on national accounts, balance of payments and trade, housing and finance, labour force consumer price indices.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/oVrC7B3n_Ms" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/ausmacrodata/Forecasting practitioner talks at ISF 2017
http://feedproxy.google.com/~r/Hyndsight/~3/0VDi4sO2V0w/
Thu, 02 Feb 2017 23:55:02 +0000https://robjhyndman.com/hyndsight/isf2017-practitioners/The International Symposium on Forecasting is a little unusual for an academic conference in that it has always had a strong presence of forecasters working in business and industry as well as academic forecasters, mostly at universities. We value the combination and interaction as it helps the academics understand the sorts of problems facing forecasters in practice, and it helps practitioners stay abreast of new methods and developments coming out of forecasting research.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/0VDi4sO2V0w" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/isf2017-practitioners/Simulating from a specified seasonal ARIMA model
http://feedproxy.google.com/~r/Hyndsight/~3/9P3jElMtGw4/
Fri, 27 Jan 2017 05:13:22 +0000https://robjhyndman.com/hyndsight/simulating-from-a-specified-seasonal-arima-model/From my email today
You use an illustration of a seasonal arima model:
ARIMA(1,1,1)(1,1,1)4
I would like to simulate data from this process then fit a model… but I am unable to find any information as to how this can be conducted… if I set phi1, Phi1, theta1, and Theta1 it would be reassuring that for large n the parameters returned by Arima(foo,order=c(1,1,1),seasonal=c(1,1,1)) are in agreement…
My answer: Unfortunately arima.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/9P3jElMtGw4" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/simulating-from-a-specified-seasonal-arima-model/IJF Tao Hong Award for the best paper in energy forecasting 2013-2014
http://feedproxy.google.com/~r/Hyndsight/~3/-GmYoRcVl70/
Wed, 11 Jan 2017 04:38:19 +0000https://robjhyndman.com/hyndsight/ijf-hong-award/Professor Tao Hong has generously funded a new prize for the best IJF paper on energy forecasting, to be awarded every two years. The first award will be for papers published in the International Journal of Forecasting during the period 2013-2014. The prize will be US$1000 plus an engraved plaque. The award committee is Rob J Hyndman, Pierre Pinson and James Mitchell.
Nominations are invited from any reader of the IJF.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/-GmYoRcVl70" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/ijf-hong-award/What is going on?
http://feedproxy.google.com/~r/Hyndsight/~3/kpeq4FvBe64/
Mon, 12 Dec 2016 02:15:50 +0000https://robjhyndman.com/hyndsight/ijf-reject/I seem to be getting an increasing number of submissions where the author has clearly not bothered to actually check that the paper was submitted correctly. Here is a rejection letter I wrote today.
Dear xxxxx
I am writing concerning manuscript #INTFOR_16xxxxx entitled “xxxxxxxxxxxxxxxx” which you submitted to the International Journal of Forecasting.
Thank you for this submission, but as it consists entirely of the IJF author guidelines, it is not suitable for publication in the IJF.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/kpeq4FvBe64" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/ijf-reject/Cross-validation for time series
http://feedproxy.google.com/~r/Hyndsight/~3/J0ePiLapqq0/
Mon, 05 Dec 2016 12:43:57 +0000https://robjhyndman.com/hyndsight/tscv/I’ve added a couple of new functions to the forecast package for R which implement two types of cross-validation for time series.K-fold cross-validation for autoregression The first is regular k-fold cross-validation for autoregressive models. Although cross-validation is sometimes not valid for time series models, it does work for autoregressions, which includes many machine learning approaches to time series. The theoretical background is provided in Bergmeir, Hyndman and Koo (2015). So cross-validation can be applied to any model where the predictors are lagged values of the response variable.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/J0ePiLapqq0" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/tscv/Invited sessions at the International Symposium on Forecasting
http://feedproxy.google.com/~r/Hyndsight/~3/GAxWra7JRlM/
Mon, 28 Nov 2016 06:55:13 +0000https://robjhyndman.com/hyndsight/isf-invited-sessions/We are currently calling for invited session proposals for the ISF to be held in Cairns, Australia, in June 2017.
An invited session consists of 3 or 4 talks around a specific forecasting theme. You are allowed to be one of the speakers in a session you organize (although it is not necessary). So if you know what you are planning to speak about, all you need to do is find 2 or 3 other speakers who will speak on something related, and invite them to join you.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/GAxWra7JRlM" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/isf-invited-sessions/We're still hiring at Monash
http://feedproxy.google.com/~r/Hyndsight/~3/si26PMWEaNw/
Sun, 06 Nov 2016 22:16:43 +0000https://robjhyndman.com/hyndsight/still-hiring-at-monash/We have another position available, this time for a lecturer (equivalent to an assistant professor tenure track in the US). The department covers a wide range of areas in statistics and econometrics, but for this position we are looking for someone with expertise in at least one of business analytics, data science, actuarial science, computational statistics and machine learning. Applicants who have recently completed a PhD, or expect to do so in the next 6 months, are welcome to apply.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/si26PMWEaNw" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/still-hiring-at-monash/Q&A: predictive analytics
http://feedproxy.google.com/~r/Hyndsight/~3/6in-bA9H82I/
Mon, 24 Oct 2016 12:00:25 +0000https://robjhyndman.com/hyndsight/qa-predictive-analytics/A major news outlet interviewed me on predictive analytics. Here were my responses. Data mining is not just for tech companies, in fact it can be especially useful for industries which are not typically thought of to be ‘innovative’ such as agriculture. What are some of the main industries that you think benefit from predictive analysis?
Any industry that collects data can use data mining and statistical modelling.
Agriculture is becoming a heavy user of data science methods, with data being collected on every aspect of crop or livestock health and development.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/6in-bA9H82I" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/qa-predictive-analytics/Q&A time
http://feedproxy.google.com/~r/Hyndsight/~3/O39WZizZZbo/
Sun, 23 Oct 2016 11:21:35 +0000https://robjhyndman.com/hyndsight/qa-time/Someone sent me some questions by email, and I decided to answer some of them here. How important is it that I know and understand the underlying mathematical framework to forecasting methods? I understand conceptually how most of them work, but I feel as if I may benefit from truly understanding the math.
The main benefit of understanding the mathematics behind different forecasting models is to be able to adapt the models when they don’t work well for your data.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/O39WZizZZbo" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/qa-time/Tourism forecasting competition data as an R package
http://feedproxy.google.com/~r/Hyndsight/~3/GakwFDKT4lQ/
Sat, 22 Oct 2016 11:19:31 +0000https://robjhyndman.com/hyndsight/tcomp/The data used in the tourism forecasting competition, discussed in Athanasopoulos et al (2011), have been made available in the Tcomp package for R. The objects are of the same format as for Mcomp package containing data from the M1 and M3 competitions.
Thanks to Peter Ellis for putting the package together. He has also produced a nice blog post about it.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/GakwFDKT4lQ" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/tcomp/GEFCom2017: Hierarchical Probabilistic Load Forecasting
http://feedproxy.google.com/~r/Hyndsight/~3/pgPVc1DEZx4/
Fri, 14 Oct 2016 14:44:58 +0000https://robjhyndman.com/hyndsight/gefcom2017/After the great success of the previous two energy forecasting competitions we have run (GEFCom2012 and GEFCom2014), we are holding another one, this time focused on hierarchical probabilistic load forecasting. Check out all the details over on Tao Hong’s blog.
The previous GEFComs have led to some major advances in forecasting methodology, available via IJF papers by the winning teams. I expect similar developments to arise out of this competition. Winners get to present their work in Cairns, Australia at ISEA2017.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/pgPVc1DEZx4" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/gefcom2017/Come to Melbourne, even if not to Monash
http://feedproxy.google.com/~r/Hyndsight/~3/Yls-FsMKYlw/
Wed, 12 Oct 2016 08:00:47 +0000https://robjhyndman.com/hyndsight/unimelb-prof-datascience/The University of Melbourne is advertising for a “Professor in Statistics (Data Science)”. Melbourne (the city) is fast becoming a vibrant centre for data science and applied statistics, with more than 4700 people signed up for the Data Science Meetup Group, a thriving start-up scene, the group at Monash Business School (including Di Cook and me), and the Monash Centre for Data Science (including Geoff Webb and Wray Buntine). Not to mention that Melbourne is a wonderful place to live, having won the “World’s most liveable city” award from the Economist for the last 6 years in a row.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Yls-FsMKYlw" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/unimelb-prof-datascience/Hadley Wickham Master R Developer course coming to Melbourne
http://feedproxy.google.com/~r/Hyndsight/~3/gDhUCEsM-NI/
Wed, 05 Oct 2016 19:53:39 +0000https://robjhyndman.com/hyndsight/hadley-course-dec-2016/Hadley Wickham’s popular R developer course is coming to Melbourne on 12-13 December 2016. Bookings can be made via Eventbrite.Hadley, of course, is the developer of the wonderful tidyverse set of R packages including ggplot2, dplyr, tidyr, readr, purrr, tibble, and many more. He is the author of several books including the new “R for Data Science”, he is the chief scientist at RStudio, and a fellow cocktail enthusiast.
From the course blurb:<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/gDhUCEsM-NI" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/hadley-course-dec-2016/Call for forecasting workshops in Cairns, Australia
http://feedproxy.google.com/~r/Hyndsight/~3/Ez_liCKtIqk/
Wed, 28 Sep 2016 15:25:44 +0000https://robjhyndman.com/hyndsight/cairns-workshops/The 37th annual International Symposium on Forecasting will be held in Cairns, Australia, from 25-28 June 2017. We plan to hold some workshops on Sunday 25 June, before the main conference.We are currently calling for workshop proposals. Proposals can be for a half-day or full-day workshop. As usual, there will be a Practitioner Track at the conference which generally leads to excellent attendance at the workshops.
Though June sounds far away, planning for the conference is well underway!<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Ez_liCKtIqk" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/cairns-workshops/Eindhoven seminar on time series visualization
http://feedproxy.google.com/~r/Hyndsight/~3/hRZKg1yp3Ew/
Tue, 27 Sep 2016 08:34:26 +0000https://robjhyndman.com/hyndsight/eindhoven-seminar/I’m currently in the Netherlands for a few weeks, and I’ll be giving a seminar at the Data Science Centre in Eindhoven next Wednesday afternoon on “Visualization of big time series data”. Details follow.Date: 5 October 2016 Time: 12.30-13.30 Venue: Filmhuis, De Zwarte Doos, 2 Den Dolech, Eindhoven
Registration is required but free. Please book here.
Abstract:
It is becoming increasingly common for organizations to collect very large amounts of data over time.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/hRZKg1yp3Ew" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/eindhoven-seminar/Forecast intervals for aggregates
http://feedproxy.google.com/~r/Hyndsight/~3/f6CtXg9vjkc/
Wed, 14 Sep 2016 14:44:50 +0000https://robjhyndman.com/hyndsight/forecast-intervals-for-aggregates/A common problem is to forecast the aggregate of several time periods of data, using a model fitted to the disaggregated data. For example, you may have monthly data but wish to forecast the total for the next year. Or you may have weekly data, and want to forecast the total for the next four weeks.
If the point forecasts are means, then adding them up will give a good estimate of the total.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/f6CtXg9vjkc" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/forecast-intervals-for-aggregates/R package forecast v7.2 now on CRAN
http://feedproxy.google.com/~r/Hyndsight/~3/I4SyFEsgCdU/
Fri, 09 Sep 2016 10:01:03 +0000https://robjhyndman.com/hyndsight/forecast-v7-2/I’ve pushed a minor update to the forecast package to CRAN. Some highlights are listed here.
Plotting time series with ggplot2 You can now facet a time series plot like this:
library(forecast) library(ggplot2) lungDeaths <- cbind(mdeaths, fdeaths) autoplot(lungDeaths, facets=TRUE) So autoplot.mts now behaves similarly to plot.mts
Multi-step fitted values The fitted function has a new argument h to allow computation of in-sample fitted values of more than one-step-ahead. In time series, fitted values are defined as the one-step-forecasts of the data used in training a model.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/I4SyFEsgCdU" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/forecast-v7-2/R packages for forecast combinations
http://feedproxy.google.com/~r/Hyndsight/~3/LngorOPSlK8/
Thu, 01 Sep 2016 09:56:16 +0000https://robjhyndman.com/hyndsight/forecast-combinations/It has been well-known since at least 1969, when Bates and Granger wrote their famous paper on “The Combination of Forecasts”, that combining forecasts often leads to better forecast accuracy.
So it is helpful to have a couple of new R packages which do just that: opera and forecastHybrid.
opera Opera stands for “Online Prediction by ExpeRt Aggregation”. It was written by Pierre Gaillard and Yannig Goude, and Pierre provides a nice introduction in the vignette.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/LngorOPSlK8" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/forecast-combinations/Sponsorship for the Cairns forecasting conference
http://feedproxy.google.com/~r/Hyndsight/~3/cGUkiHJCozI/
Wed, 31 Aug 2016 02:07:07 +0000https://robjhyndman.com/hyndsight/isf2017-sponsorship/Regular readers will know that the International Symposium on Forecasting is coming to Australia in June 2017. This is the leading international forecasting conference, and one I’ve attended every year for the past 17 years.
It will be held in Cairns, Australia — one of the most beautiful locations in the country (and there is some stiff competition!) and right next to the Great Barrier Reef. Some further information is available on our website (still in progress).<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/cGUkiHJCozI" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/isf2017-sponsorship/Rmarkdown template for a Monash working paper
http://feedproxy.google.com/~r/Hyndsight/~3/GO7zr3iYhYQ/
Sun, 28 Aug 2016 10:21:58 +0000https://robjhyndman.com/hyndsight/rmarkdown-template/This is only directly relevant to my Monash students and colleagues, but the same idea might be useful for adapting to other institutions.
Some recent changes in the rmarkdown and bookdown packages mean that it is now possible to produce working papers in exactly the same format as we previously used with LaTeX.The following template will do the trick. Notice the figure labeling and referencing, and the citations using @key.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/GO7zr3iYhYQ" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/rmarkdown-template/The thief package for R: Temporal HIErarchical Forecasting
http://feedproxy.google.com/~r/Hyndsight/~3/WnufzXUu3ak/
Mon, 22 Aug 2016 07:58:42 +0000https://robjhyndman.com/hyndsight/thief/I have a new R package available to do temporal hierarchical forecasting, based on my paper with George Athanasopoulos, Nikolaos Kourentzes and Fotios Petropoulos. (Guess the odd guy out there!)
It is called “thief” - an acronym for Temporal HIErarchical Forecasting. The idea is to take a seasonal time series, and compute all possible temporal aggregations that result in an integer number of observations per year. For example, a quarterly time series is aggregated to biannual and annual; while a monthly time series is aggregated to 2-monthly, quarterly, 4-monthly, biannual and annual.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/WnufzXUu3ak" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/thief/"Forecasting with R" short course in Eindhoven
http://feedproxy.google.com/~r/Hyndsight/~3/d-eN6bMx9-M/
Thu, 18 Aug 2016 00:09:47 +0000https://robjhyndman.com/hyndsight/eindhoven-course/I will be giving my 3-day short-course/workshop on “Forecasting with R” in Eindhoven (Netherlands) from 19-21 October.
Details at https://www.win.tue.nl/~adriemel/shortcourse.html
Register here<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/d-eN6bMx9-M" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/eindhoven-course/Tourism time series repository
http://feedproxy.google.com/~r/Hyndsight/~3/_pVyNFo2E1I/
Wed, 10 Aug 2016 08:12:44 +0000https://robjhyndman.com/hyndsight/tourism-time-series/A few years ago, I wrote a paper with George Athanasopoulos and others about a tourism forecasting competition. We originally made the data available as an online supplement to the paper, but that has unfortunately since disappeared although the paper itself is still available.
So I am posting the data here in case anyone wants to use it for replicating our results, or for other research purposes. The data are split into monthly, quarterly and yearly data.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/_pVyNFo2E1I" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/tourism-time-series/Statistics positions available at Monash University
http://feedproxy.google.com/~r/Hyndsight/~3/0gWzJpjoD4c/
Wed, 27 Jul 2016 23:52:37 +0000https://robjhyndman.com/hyndsight/jobs2017/We are hiring again, and looking for people in statistics, econometrics and related fields (such as actuarial science, machine learning, and business analytics). We have a strong business analytics group (with particular expertise in data visualization, machine learning, statistical computing, R, and forecasting), and it would be great to see it grow. The official advert follows.
The Department of Econometrics and Business Statistics at Monash Business School in Melbourne, Australia, invites applications for full-time tenure-track positions at the Senior Lecturer level (equivalent to North American/European Assistant Professor with some post-doctoral academic experience) and Associate Professor level.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/0gWzJpjoD4c" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/jobs2017/The latest IJF issue with GEFCom2014 results
http://feedproxy.google.com/~r/Hyndsight/~3/KAeVP0qhp4M/
Fri, 10 Jun 2016 01:07:09 +0000https://robjhyndman.com/hyndsight/ijf-gefcom2014/The latest issue of the IJF is a bumper issue with over 500 pages of forecasting insights.
The GEFCom2014 papers are included in a special section on probabilistic energy forecasting, guest edited by Tao Hong and Pierre Pinson. This is a major milestone in energy forecasting research with the focus on probabilistic forecasting and forecast evaluation done using a quantile scoring method. Only a few years ago I was having to explain to energy professionals why you couldn’t use a MAPE to evaluate a percentile forecast.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/KAeVP0qhp4M" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/ijf-gefcom2014/2017 International Symposium on Energy Analytics
http://feedproxy.google.com/~r/Hyndsight/~3/G1rjDJDRfxg/
Fri, 03 Jun 2016 02:56:36 +0000https://robjhyndman.com/hyndsight/isea2017/Predictive Energy Analytics in the Big Data World Cairns, Australia, June 22-23, 2017 ISEA2017
This will be a great conference, and it is in a great location — Cairns, Australia, right by the Great Barrier Reef. Even better, if you stay on you can attend the International Symposium on Forecasting which immediately follows the International Symposium on Energy Analytics.
So block out 22-28 June 2017 on your calendars so you can enjoy a tropical paradise in one of the most beautiful parts of Australia, while attending two awesome conferences.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/G1rjDJDRfxg" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/isea2017/Forecast v7 (part 2)
http://feedproxy.google.com/~r/Hyndsight/~3/NwDB_EJ3bOc/
Wed, 01 Jun 2016 07:22:52 +0000https://robjhyndman.com/hyndsight/forecast7-part-2/As mentioned in my previous post on the forecast package v7, the most visible feature was the introduction of ggplot2 graphics. This post briefly summarizes the remaining new features of forecast v7. library(forecast) library(ggplot2) tslm rewritten The tslm function is designed to fit linear models to time series data. It is intended to approximately mimic lm (and calls lm to do the estimation), but to package the output to remember the ts attributes.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/NwDB_EJ3bOc" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/forecast7-part-2/Explore Australian Elections Data with R
http://feedproxy.google.com/~r/Hyndsight/~3/UH2nTM8mr4w/
Fri, 27 May 2016 04:07:11 +0000https://robjhyndman.com/hyndsight/eechidna/This is a guest post by my colleague Professor Di Cook, cross-posted from her Visiphilia blog. Di and I are two of the authors of the new eechidna package for R, now on CRAN.The eechidna package has just been posted on CRAN, in time for the longest election campaign in Australia since the 1950s. The next Federal election scheduled for 2nd July was announced a few weeks ago. A little before this a handful of academics met at the first ever Australian ROpenSci Unnconference at the Microsoft headquarters in Brisbane in an event primarily organised by students at the Queensland University of Technology.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/UH2nTM8mr4w" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/eechidna/SSA helping you find a job
http://feedproxy.google.com/~r/Hyndsight/~3/LSDoF5calzw/
Thu, 12 May 2016 02:04:00 +0000https://robjhyndman.com/hyndsight/ssa-jobs-board/One of the great services of the Statistical Society of Australia is an excellent jobs board advertising available jobs for statisticians, data analysts, data scientists, etc. Jobs can be filtered by industry, location and job function.
Today the SSA announced a new service to job seekers: CV/Resume Critique. As a job seeker, registered on the SSA Job Board, you now have the option to request a free, confidential CV/resume evaluation from an expert and writer.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/LSDoF5calzw" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/ssa-jobs-board/forecast v7 and ggplot2 graphics
http://feedproxy.google.com/~r/Hyndsight/~3/79WsF98TPYQ/
Mon, 09 May 2016 05:00:00 +0000https://robjhyndman.com/hyndsight/forecast7-ggplot2/Version 7 of the forecast package was released on CRAN about a month ago, but I’m only just getting around to posting about the new features.
The most visible feature was the introduction of ggplot2 graphics. I first wrote the forecast package before ggplot2 existed, and so only base graphics were available. But I figured it was time to modernize and use the nice features available from ggplot2. The following examples illustrate the main new graphical functionality.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/79WsF98TPYQ" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/forecast7-ggplot2/Melbourne Data Science Initiative 2016
http://feedproxy.google.com/~r/Hyndsight/~3/QV2qt-qV7_A/
Wed, 13 Apr 2016 03:51:28 +0000https://robjhyndman.com/hyndsight/medascin2016/In just over three weeks, the inaugural MeDaScIn event will take place. This is an initiative to grow the talent pool of local data scientists and to promote Melbourne as a world city of excellence in Data Science.
The main event takes place on Friday 6th May, with lots of interesting sounding titles and speakers from business and government. I’m the only academic speaker on the program, giving the closing talk on “Automatic FoRecasting”.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/QV2qt-qV7_A" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/medascin2016/Sample quantiles 20 years later
http://feedproxy.google.com/~r/Hyndsight/~3/_VJKF2Y2UyQ/
Mon, 28 Mar 2016 04:35:17 +0000https://robjhyndman.com/hyndsight/sample-quantiles-20-years-later/Almost exactly 20 years ago I wrote a paper with Yanan Fan on how sample quantiles are computed in statistical software. It was cited 43 times in the first 10 years, and 457 times in the next 10 years, making it my third paper to receive 500+ citations.
So what happened in 2006 to suddenly increase the citations? I think it was a combination of things: I wrote a new quantile() function (with Ivan Frohne) which made it into R core v2.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/_VJKF2Y2UyQ" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/sample-quantiles-20-years-later/Plotting overlapping prediction intervals
http://feedproxy.google.com/~r/Hyndsight/~3/xWKG2WfnQBU/
Wed, 23 Mar 2016 10:19:49 +0000https://robjhyndman.com/hyndsight/overlappingpi/I often see figures with two sets of prediction intervals plotted on the same graph using different line types to distinguish them. The results are almost always unreadable. A better way to do this is to use semi-transparent shaded regions. Here is an example showing two sets of forecasts for the Nile River flow.
library(forecast) f1 = forecast(auto.arima(Nile, lambda=0), h=20, level=95) f2 = forecast(ets(Nile), h=20, level=95) plot(f1, shadecol=rgb(0,0,1,.4), flwd=1, main="Forecasts of Nile River flow", xlab="Year", ylab="Billions of cubic metres") polygon(c(time(f2$mean),rev(time(f2$mean))), c(f2$lower,rev(f2$upper)), col=rgb(1,0,0,.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/xWKG2WfnQBU" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/overlappingpi/rOpenSci unconference in Brisbane, 21-22 April 2016
http://feedproxy.google.com/~r/Hyndsight/~3/Runud4Y_Jyc/
Mon, 21 Mar 2016 00:31:07 +0000https://robjhyndman.com/hyndsight/ropensci2016/The first rOpenSci unconference in Australia will be held on Thursday and Friday (April 21-22) in Brisbane, at the Microsoft Innovation Centre.
This event will bring together researchers, developers, data scientists and open data enthusiasts from industry, government and university. The aim is to conceptualise and develop R-based tools that address current challenges in data science, open science and reproducibility.
Past examples of the projects can here, here, and here. Also here.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Runud4Y_Jyc" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/ropensci2016/Monash Business Analytics Team Profile
http://feedproxy.google.com/~r/Hyndsight/~3/zUTPJkOVDco/
Wed, 09 Mar 2016 07:06:16 +0000https://robjhyndman.com/hyndsight/monash-insider/Our research group been growing lately, as you can see below! We were featured in the latest issue of the Monash newsletter The Insider. Check it out.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/zUTPJkOVDco" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/monash-insider/Model variance for ARIMA models
http://feedproxy.google.com/~r/Hyndsight/~3/_BWI2i0D1o8/
Fri, 04 Mar 2016 11:15:19 +0000https://robjhyndman.com/hyndsight/model-variance-for-arima-models/From today’s email:
I wanted to ask you about your R forecast package, in particular the Arima() function. We are using this function to fit an ARIMAX model and produce model estimates and standard errors, which in turn can be used to get p-values and later model forecasts. To double check our work, we are also fitting the same model in SAS using PROC ARIMA and comparing model coefficients and output.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/_BWI2i0D1o8" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/model-variance-for-arima-models/Data Science for Managers: April 2016
http://feedproxy.google.com/~r/Hyndsight/~3/hz8m5Jir7O8/
Sun, 28 Feb 2016 23:34:12 +0000https://robjhyndman.com/hyndsight/data-science-for-managers-april-2016/We are running our short course on “Data Science for Managers” again: 11-13 April 2016. All the details are on the website.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/hz8m5Jir7O8" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/data-science-for-managers-april-2016/Omitting outliers
http://feedproxy.google.com/~r/Hyndsight/~3/XZbH7ZBGs00/
Wed, 24 Feb 2016 22:55:11 +0000https://robjhyndman.com/hyndsight/omitting-outliers/Someone sent me this email today:
One of my colleagues said that you once said/wrote that you had encountered very few real outliers in your work, and that normally the “outlier-looking” data points were proper data points that should not have been treated as outliers. Have you discussed this in writing? If so, I would love to read it.
I don’t think I’ve ever said or written anything quite like that, and I see lots of outliers in real data.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/XZbH7ZBGs00" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/omitting-outliers/Making data analysis easier: Hadley Wickham at WOMBAT2016
http://feedproxy.google.com/~r/Hyndsight/~3/Z7uFJDzy5cM/
Wed, 24 Feb 2016 22:25:44 +0000https://robjhyndman.com/hyndsight/wombat2016-hadley/Slides for Hadley’s talk
The slides for all the other talks from the workshop are also now online at wombat2016.org<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Z7uFJDzy5cM" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/wombat2016-hadley/Electricity price forecasting competition
http://feedproxy.google.com/~r/Hyndsight/~3/q9qNEncRiGc/
Wed, 17 Feb 2016 05:51:07 +0000https://robjhyndman.com/hyndsight/electricity-price-forecasting-competition/The GEFCom competitions have been a great success in generating good research on forecasting methods for electricity demand, and in enabling a comprehensive comparative evaluation of various methods. But they have only considered price forecasting in a simplified setting. So I’m happy to see this challenge is being taken up as part of the European Energy Market Conference for 2016, to be held from 6-9 June at the University of Porto in Portugal.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/q9qNEncRiGc" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/electricity-price-forecasting-competition/What's your Hall number?
http://feedproxy.google.com/~r/Hyndsight/~3/LAYa9cb7dPQ/
Thu, 28 Jan 2016 11:48:43 +0000https://robjhyndman.com/hyndsight/whats-your-hall-number/Today I attended the funeral of Peter Hall, one of the finest mathematical statisticians ever to walk the earth and easily the best from Australia. One of the most remarkable things about Peter was his astonishing productivity, with over 600 papers. As I sat in the audience I realised that many of the people there were probably coauthors of papers with Peter, and I wondered how many statisticians in the world would have been his coauthors or second-degree co-authors.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/LAYa9cb7dPQ" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/whats-your-hall-number/ACEMS Business Analytics Prize 2016
http://feedproxy.google.com/~r/Hyndsight/~3/rHHC2uZuCcQ/
Wed, 20 Jan 2016 05:03:37 +0000https://robjhyndman.com/hyndsight/acems-prize-2016/We have established a new annual prize for research students at Monash University in the general area of business analytics, funded by the Australian Centre of Excellence in Mathematical and Statistical Frontiers (ACEMS). The rules of the award are listed below.
The student must have submitted a paper to a high quality journal or refereed conference on some topic in the general area of business analytics, computational statistics or data visualization.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/rHHC2uZuCcQ" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/acems-prize-2016/Farewell Peter Hall (1951-2016)
http://feedproxy.google.com/~r/Hyndsight/~3/mPt659ncQqY/
Mon, 11 Jan 2016 05:12:37 +0000https://robjhyndman.com/hyndsight/farewell-peter-hall-1951-2016/Peter Hall passed away on Saturday after a long battle with illness over the last couple of years. No statistician will need reminding of Peter’s extensive contributions to the field. He had over 500 published papers, and had won every major award available, many of them listed on his Wikipedia page.I spent a few months visiting Peter at ANU in 2002, and we wrote a couple of papers together (here and here).<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/mPt659ncQqY" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/farewell-peter-hall-1951-2016/Starting a career in data science
http://feedproxy.google.com/~r/Hyndsight/~3/QX6HhFG6LCk/
Wed, 30 Dec 2015 04:29:45 +0000https://robjhyndman.com/hyndsight/starting-a-career-in-data-science/I received this email from one of my undergraduate students:
I’m writing to you asking for advice on how to start a career in Data Science. Other professions seem a bit more straight forward, in that accountants for example simply look for Internships and ways into companies from there. From my understanding, the nature of careers in data science seem to be on a project-to-project basis. I’m not sure how to get my foot stuck in the door.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/QX6HhFG6LCk" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/starting-a-career-in-data-science/Making data analysis easier
http://feedproxy.google.com/~r/Hyndsight/~3/of5723OqL6c/
Sun, 20 Dec 2015 23:46:29 +0000https://robjhyndman.com/hyndsight/wombat2016/Di Cook and I are organizing a workshop on “Making data analysis easier” for 18-19 February 2016.
We are calling it WOMBAT2016, which an acronym for Workshop Organized by the Monash Business Analytics Team. Appropriately, it will be held at the Melbourne Zoo. Our plan is to make these workshops an annual event.
Some details are available on the workshop website. Key features are:
Hadley Wickham is our keynote speaker.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/of5723OqL6c" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/wombat2016/RStudio just keeps getting better
http://feedproxy.google.com/~r/Hyndsight/~3/K9enKl-drNA/
Fri, 11 Dec 2015 05:36:02 +0000https://robjhyndman.com/hyndsight/rstudio-just-keeps-getting-better/RStudio has been a life-changer for the way I work, and for how I teach data analysis. I still have a couple of minor frustrations with it, but they are slowly disappearing as RStudio adds features.
I use dual monitors and I like to code on one monitor and have the console and plots on the other monitor. Otherwise I see too little context, and long lines get wrapped making the code harder to read.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/K9enKl-drNA" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/rstudio-just-keeps-getting-better/Who's downloading the forecast package?
http://feedproxy.google.com/~r/Hyndsight/~3/Z_sz5IslBYo/
Wed, 09 Dec 2015 21:27:30 +0000https://robjhyndman.com/hyndsight/fpp-downloads/The github page for the forecast package currently shows the following information
Note the downloads figure: 264K/month. I know the package is popular, but that seems crazy. Also, the downloads figure on github only counts the downloads from the RStudio mirror, and ignores downloads from the other 125 mirrors around the world.Here are the top ten downloaded packages from the last month:
library(cranlogs) cran_top_downloads(when='last-month') rank package count from to 1 zoo 308290 2015-11-09 2015-12-08 2 forecast 263797 2015-11-09 2015-12-08 3 Rcpp 260636 2015-11-09 2015-12-08 4 lmtest 258810 2015-11-09 2015-12-08 5 fpp 244989 2015-11-09 2015-12-08 6 expsmooth 244179 2015-11-09 2015-12-08 7 fma 243556 2015-11-09 2015-12-08 8 tseries 243172 2015-11-09 2015-12-08 9 stringi 199384 2015-11-09 2015-12-08 10 ggplot2 192072 2015-11-09 2015-12-08 OK, that is very weird.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Z_sz5IslBYo" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/fpp-downloads/The hidden benefits of open-source software
http://feedproxy.google.com/~r/Hyndsight/~3/AEXZdIqJbOw/
Sun, 29 Nov 2015 23:22:00 +0000https://robjhyndman.com/hyndsight/oss-benefits/I’ve been having discussions with colleagues and university administration about the best way for universities to manage home-grown software.
The traditional business model for software is that we build software and sell it to everyone willing to pay. Very often, that leads to a software company spin-off that has little or nothing to do with the university that nurtured the development. Think MATLAB, S-Plus, Minitab, SAS and SPSS, all of which grew out of universities or research institutions.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/AEXZdIqJbOw" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/oss-benefits/ODI looking for young postgrad statisticians
http://feedproxy.google.com/~r/Hyndsight/~3/c8CFwdq17HA/
Fri, 13 Nov 2015 00:26:03 +0000https://robjhyndman.com/hyndsight/odi-fellowships/The Overseas Development Institute Fellowship Scheme sends young postgraduate statisticians (and economists) to work in the public sectors of developing countries in Africa, the Caribbean and the Pacific on two-year contracts. This is a great way to develop skills and gain experience working within a developing country’s government. And you get to live in a fascinating place!
The application process for the 2016-2018 Fellowship Scheme is now open. Students are advised to apply before 17 December 2015 for a chance to be part of the ODI Fellowship Scheme.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/c8CFwdq17HA" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/odi-fellowships/Big Data for Official Statistics Competition
http://feedproxy.google.com/~r/Hyndsight/~3/TsdBVfv_m9o/
Thu, 12 Nov 2015 00:01:43 +0000https://robjhyndman.com/hyndsight/bdcomp/This is a new competition being organized by EuroStat. The first phase involves nowcasting economic indicators at national and European level including unemployment, HICP, Tourism and Retail Trade and some of their variants.
The main goal of the competition is to discover promising methodologies and data sources that could, now or in the future, be used to improve the production of official statistics in the European Statistical System.
The organizers seem to have been encouraged by the success of Kaggle and other data science competition platforms.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/TsdBVfv_m9o" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/bdcomp/Jobs at Monash University
http://feedproxy.google.com/~r/Hyndsight/~3/QJXsIRm4B1E/
Fri, 06 Nov 2015 06:05:17 +0000https://robjhyndman.com/hyndsight/jobs2016/We have two new continuing positions currently being advertised in our department: for lecturer and senior lecturer. Details are on the Monash website. (For those in North America, a lecturer is equivalent to your assistant professor, and a senior lecturer is equivalent to your associate professor. See the Wikipedia article on Australian academic ranks for more information.)
Although the title says “Lecturer/Senior Lecturer (Econometrics)”, we are interested in a wider range of candidates including statistics and machine learning.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/QJXsIRm4B1E" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/jobs2016/Piecewise linear trends
http://feedproxy.google.com/~r/Hyndsight/~3/-WriXyxvM-o/
Wed, 28 Oct 2015 02:43:47 +0000https://robjhyndman.com/hyndsight/piecewise-linear-trends/I prepared the following notes for a consulting client, and I thought they might be of interest to some other people too.
Let \(y_t\) denote the value of the time series at time \(t\), and suppose we wish to fit a trend with correlated errors of the form \[ y_t = f(t) + n_t, \]
where \(f(t)\) represents the possibly nonlinear trend and \(n_t\) is an autocorrelated error process.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/-WriXyxvM-o" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/piecewise-linear-trends/forecast package v6.2
http://feedproxy.google.com/~r/Hyndsight/~3/ErkEbj1dht8/
Tue, 20 Oct 2015 06:39:21 +0000https://robjhyndman.com/hyndsight/forecast6-2/It is a while since I last updated the CRAN version of the forecast package, so I uploaded the latest version (6.2) today. The github version remains the most up-to-date version and is already two commits ahead of the CRAN version.
This update is mostly bug fixes and additional error traps. The full ChangeLog is listed below. Many unit tests added using testthat.
Fixed bug in ets() when very short seasonal series were passed in a data frame.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/ErkEbj1dht8" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/forecast6-2/Stanford seminar
http://feedproxy.google.com/~r/Hyndsight/~3/qMYT5iV34BI/
Wed, 07 Oct 2015 06:09:16 +0000https://robjhyndman.com/hyndsight/stanford-seminar/I gave a seminar at Stanford today. Slides are below. It was definitely the most intimidating audience I’ve faced, with Jerome Friedman, Trevor Hastie, Brad Efron, Persi Diaconis, Susan Holmes, David Donoho and John Chambers all present (and probably other famous names I’ve missed).
I’ll be giving essentially the same talk at UC Davis on Thursday.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/qMYT5iV34BI" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/stanford-seminar/Reproducibility in computational research
http://feedproxy.google.com/~r/Hyndsight/~3/YrjKQefQ4ro/
Fri, 25 Sep 2015 07:31:05 +0000https://robjhyndman.com/hyndsight/reproducibility/Jane Frazier spoke at our research team meeting today on “Reproducibility in computational research”. We had a very stimulating and lively discussion about the issues involved. One interesting idea was that reproducibility is on a scale, and we can all aim to move further along the scale towards making our own research more reproducible. For example
Can you reproduce your results tomorrow on the same computer with the same software installed?<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/YrjKQefQ4ro" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/reproducibility/Chinese R conference
http://feedproxy.google.com/~r/Hyndsight/~3/iL9mMzFnW24/
Thu, 24 Sep 2015 06:42:36 +0000https://robjhyndman.com/hyndsight/chinese-r-conference/I will be speaking at the Chinese R conference in Nanchang, to be held on 24-25 October, on “Forecasting Big Time Series Data using R”.
Details (for those who can read Chinese) are at china-r.org.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/iL9mMzFnW24" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/chinese-r-conference/Upcoming talks in California
http://feedproxy.google.com/~r/Hyndsight/~3/2xdIBUwPukI/
Tue, 22 Sep 2015 06:41:42 +0000https://robjhyndman.com/hyndsight/upcoming-talks-in-california/I’m back in California for the next couple of weeks, and will give the following talk at Stanford and UC-Davis.
Optimal forecast reconciliation for big time series data Time series can often be naturally disaggregated in a hierarchical or grouped structure. For example, a manufacturing company can disaggregate total demand for their products by country of sale, retail outlet, product type, package size, and so on. As a result, there can be millions of individual time series to forecast at the most disaggregated level, plus additional series to forecast at higher levels of aggregation.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/2xdIBUwPukI" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/upcoming-talks-in-california/International Symposium on Forecasting: Spain 2016
http://feedproxy.google.com/~r/Hyndsight/~3/n2UmnYzsNDk/
Mon, 21 Sep 2015 09:55:15 +0000https://robjhyndman.com/hyndsight/isf2016/June 19-22, 2016 Santander, Spain – Palace of La Magdalena
The International Symposium on Forecasting (ISF) is the premier forecasting conference, attracting the world’s leading forecasting researchers, practitioners, and students. Through a combination of keynote speaker presentations, academic sessions, workshops, and social programs, the ISF provides many excellent opportunities for networking, learning, and fun.
Speakers: Greg Allenby, The Ohio State University, USA Todd Clark, Federal Reserve Bank of Cleveland, USA José Duato, Polytechnic University of Valencia, Spain Robert Fildes, Lancaster University, United Kingdom Edward Leamer, UCLA Anderson, USA Henrik Madsen, Technical University of Denmark Adrian Raftery, University of Washington, USA<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/n2UmnYzsNDk" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/isf2016/Unbelievable
http://feedproxy.google.com/~r/Hyndsight/~3/v55_YNUKjD4/
Tue, 15 Sep 2015 23:36:36 +0000https://robjhyndman.com/hyndsight/unbelievable/This is a very different book from my usual areas of forecasting and statistics. It is a personal memoir describing my journey of deconversion from Christianity.
Until a few years ago, I was regularly speaking at church conferences internationally, and my books are still used in Bible classes and Sunday Schools around the world. I even helped establish an innovative new church, which became a model for similar churches in other countries.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/v55_YNUKjD4" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/unbelievable/IJF vol 31(4): Forecasting in telecommunications and ICT
http://feedproxy.google.com/~r/Hyndsight/~3/s8Gk1UIoFgw/
Tue, 15 Sep 2015 09:29:47 +0000https://robjhyndman.com/hyndsight/ijf-vol-314/The last issue of the International Journal of Forecasting for 2015 has been released. This one contains the usual mix of topics, plus a special section on Forecasting in telecommunications and ICT including a nice review article by Nigel Meade and Towhidul Islam. Enjoy!<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/s8Gk1UIoFgw" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/ijf-vol-314/Advice to other journal editors
http://feedproxy.google.com/~r/Hyndsight/~3/wb0VzByS0BQ/
Sun, 13 Sep 2015 10:50:16 +0000https://robjhyndman.com/hyndsight/finding-reviewers/I get asked to review journal papers almost every day, and I have to say no to almost all of them. I know it is hard to find reviewers, but many of these requests indicate very lazy editors. So to all the editors out there looking for reviewers, here is some advice.
Never ask someone who is an editor for another journal. I am handling about 500 submissions per year for the International Journal of Forecasting, and about 10 per year for the Journal of Statistical Software.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/wb0VzByS0BQ" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/finding-reviewers/Mathematical annotations on R plots
http://feedproxy.google.com/~r/Hyndsight/~3/sZM5wSV_aOI/
Wed, 02 Sep 2015 07:17:08 +0000https://robjhyndman.com/hyndsight/latex2exp/I’ve always struggled with using plotmath via the expression function in R for adding mathematical notation to axes or legends. For some reason, the most obvious way to write something never seems to work for me and I end up using trial and error in a loop with far too many iterations.
So I am very happy to see the new latex2exp package available which translates LaTeX expressions into a form suitable for R graphs.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/sZM5wSV_aOI" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/latex2exp/Data Science for Managers (short course)
http://feedproxy.google.com/~r/Hyndsight/~3/qwaUppXL6qI/
Thu, 27 Aug 2015 07:45:07 +0000https://robjhyndman.com/hyndsight/data-science-for-managers-2015/I am teaching part of a short-course on Data Science for Managers from 10-12 October in Melbourne.
Course Overview The impact of Data Science on modern business is second only to the introduction of computers. And yet, for many businesses the barrier of entry remains too high due to lack of knowhow, organisational inertia, difficulties in hiring the right manpower, an apparent need for upfront commitment, and more.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/qwaUppXL6qI" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/data-science-for-managers-2015/The bias-variance decomposition
http://feedproxy.google.com/~r/Hyndsight/~3/E9zG8k9vU7c/
Mon, 03 Aug 2015 07:59:04 +0000https://robjhyndman.com/hyndsight/bias-variance/This week, I am teaching my Business Analytics class about the bias-variance trade-off. For some reason, the proof is not contained in either ESL or ISL, even though it is quite simple. I also discovered that the proof currently provided on Wikipedia makes little sense in places.
So I wrote my own for the class. It is longer than necessary to ensure there are no jumps that might confuse students.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/E9zG8k9vU7c" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/bias-variance/Murphy diagrams in R
http://feedproxy.google.com/~r/Hyndsight/~3/0T-nVYZNeHE/
Thu, 16 Jul 2015 07:57:46 +0000https://robjhyndman.com/hyndsight/murphy-diagrams/At the recent International Symposium on Forecasting, held in Riverside, California, Tillman Gneiting gave a great talk on “Evaluating forecasts: why proper scoring rules and consistent scoring functions matter”. It will be the subject of an IJF invited paper in due course.
One of the things he talked about was the “Murphy diagram” for comparing forecasts, as proposed in Ehm et al (2015). Here’s how it works for comparing mean forecasts.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/0T-nVYZNeHE" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/murphy-diagrams/Useful tutorials
http://feedproxy.google.com/~r/Hyndsight/~3/-jPVOOJ_I70/
Wed, 01 Jul 2015 03:28:18 +0000https://robjhyndman.com/hyndsight/useful-tutorials/There are some tools that I use regularly, and I would like my research students and post-docs to learn them too. Here are some great online tutorials that might help.
ggplot tutorial from Winston Chang
Writing an R package from Karl Broman
Rmarkdown from RStudio
Shiny from RStudio
git/github guide from Karl Broman
minimal make tutorial from Karl Broman<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/-jPVOOJ_I70" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/useful-tutorials/My Yahoo talk is now online
http://feedproxy.google.com/~r/Hyndsight/~3/gqogtveYPK0/
Mon, 29 Jun 2015 17:12:19 +0000https://robjhyndman.com/hyndsight/yahoo2015/Last week I gave a talk in the Yahoo! Big Thinkers series. The video of the talk is now online and embedded below.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/gqogtveYPK0" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/yahoo2015/Keeping up to date with my research papers
http://feedproxy.google.com/~r/Hyndsight/~3/OMeWNFJiu64/
Sat, 27 Jun 2015 05:22:17 +0000https://robjhyndman.com/hyndsight/new-papers/Many people ask me to let them know when I write a new research paper. I can’t do that as there are too many people involved, and it is not scalable.
The solution is simple. Take your pick from the following options. Each is automatic and will let you know whenever I produce a new paper.
Subscribe to the rss feed on my website using feedly or some other rss reader.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/OMeWNFJiu64" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/new-papers/IJF best paper awards
http://feedproxy.google.com/~r/Hyndsight/~3/IsOuSnCH_yw/
Tue, 23 Jun 2015 21:32:16 +0000https://robjhyndman.com/hyndsight/ijf-best-paper-awards/Today at the International Symposium on Forecasting, I announced the awards for the best paper published in the International Journal of Forecasting in the period 2012-2013.
We make an award every two years to the best paper(s) published in the journal. There is always about 18 months delay after the publication period to allow time for reflection, citations, etc. The selected papers are selected by vote of the editorial board. The best paper wins an engraved bronze plaque and US$1000.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/IsOuSnCH_yw" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/ijf-best-paper-awards/North American seminars: June 2015
http://feedproxy.google.com/~r/Hyndsight/~3/8CY5JJtY6ww/
Tue, 16 Jun 2015 03:11:43 +0000https://robjhyndman.com/hyndsight/northamerica2015/For the next few weeks I am travelling in North America and will be giving the following talks.
19 June: Southern California Edison, Rosemead CA. “Probabilistic forecasting of peak electricity demand”.
23 June: International Symposium on Forecasting, Riverside CA. “MEFM: An R package for long-term probabilistic forecasting of electricity demand”.
25 June: Google, Mountain View, CA. “Automatic algorithms for time series forecasting”.
26 June: Yahoo, Sunnyvale, CA.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/8CY5JJtY6ww" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/northamerica2015/R vs Autobox vs ForecastPro vs ...
http://feedproxy.google.com/~r/Hyndsight/~3/MibNOjcX7cc/
Wed, 03 Jun 2015 01:04:56 +0000https://robjhyndman.com/hyndsight/show-me-the-evidence/Every now and then a commercial software vendor makes claims on social media about how their software is so much better than the forecast package for R, but no details are provided.
There are lots of reasons why you might select a particular software solution, and R isn’t for everyone. But anyone claiming superiority should at least provide some evidence rather than make unsubstantiated claims.The M3 forecasting competition was organized by Spyros Makridakis and Michèle Hibon.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/MibNOjcX7cc" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/show-me-the-evidence/A new R package for detecting unusual time series
http://feedproxy.google.com/~r/Hyndsight/~3/CyVvpCn_Jas/
Sun, 31 May 2015 05:59:52 +0000https://robjhyndman.com/hyndsight/anomalous/The anomalous package provides some tools to detect unusual time series in a large collection of time series. This is joint work with Earo Wang (an honours student at Monash) and Nikolay Laptev (from Yahoo Labs). Yahoo is interested in detecting unusual patterns in server metrics. The package is based on this paper with Earo and Nikolay.
The basic idea is to measure a range of features of the time series (such as strength of seasonality, an index of spikiness, first order autocorrelation, etc.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/CyVvpCn_Jas" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/anomalous/New in forecast 6.0
http://feedproxy.google.com/~r/Hyndsight/~3/SWpJ54Rvyto/
Fri, 15 May 2015 05:01:58 +0000https://robjhyndman.com/hyndsight/forecast6/This week I uploaded a new version of the forecast package to CRAN. As there were a lot of changes, I decided to increase the version number to 6.0.
The changes are all outlined in the ChangeLog file as usual. I will highlight some of the more important changes since v5.0 here.ETS One of the most used functions in the package is ets() and it provides a stock forecasting engine for many organizations.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/SWpJ54Rvyto" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/forecast6/More changes to the IJF editorial board
http://feedproxy.google.com/~r/Hyndsight/~3/paESBguoGwI/
Thu, 14 May 2015 03:55:26 +0000https://robjhyndman.com/hyndsight/ijf-board/The editorial board of the International Journal of Forecasting is going through a renewal process with several changes to the team of editors and the team of associate editors in the last few weeks.New Editors Graham Elliott has decided to step down from the IJF editorial board after many years of service. Graham is best known for his research on optimal forecast combination, and forecasting under asymmetric and flexible loss functions.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/paESBguoGwI" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/ijf-board/Nominations for IJF Best Paper 2012-2013
http://feedproxy.google.com/~r/Hyndsight/~3/z2mgheObrPw/
Thu, 07 May 2015 07:46:49 +0000https://robjhyndman.com/hyndsight/ijf-nominations2/The following papers have been nominated for the best paper published in the International Journal of Forecasting in 2012-2013. I have included an excerpt from the nomination in each case. The papers in bold have been short-listed for the award, and the editorial board are currently voting on them. Bellotti, T., & Crook, J. (2012). Loss given default models incorporating macroeconomic variables for credit cards. IJF, 28(1), 171-182.
The first rule for the award of best paper should be that the paper clearly reflects the value of the new method/approach when compared to established alternatives in the particular problem context chosen by the researchers.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/z2mgheObrPw" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/ijf-nominations2/Thinking big at Yahoo
http://feedproxy.google.com/~r/Hyndsight/~3/2qbp80RZQzY/
Wed, 22 Apr 2015 23:16:50 +0000https://robjhyndman.com/hyndsight/thinking-big-at-yahoo/I’m speaking in the “Yahoo Labs Big Thinkers” series on Friday 26 June. I hope I can live up to the title!
My talk is on “Exploring the boundaries of predictability: what can we forecast, and when should we give up?” Essentially I will start with some of the ideas in this post, and then discuss the features of hard-to-forecast time series.
So if you’re in the San Francisco Bay area, please come along.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/2qbp80RZQzY" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/thinking-big-at-yahoo/Travelling Thilaksha
http://feedproxy.google.com/~r/Hyndsight/~3/ZzhaVCrH8LU/
Wed, 22 Apr 2015 07:23:45 +0000https://robjhyndman.com/hyndsight/travelling-thilaksha/One of my PhD students, Thilaksha Tharanganie, has been very successful in getting travel funding to attend conferences. She was the subject of a write-up in today’s Monash News.
We encourage students to attend conferences, and provide funding for them to attend one international conference and one local conference during their PhD candidature. Thilaksha was previously funded to attend last year’s COMPSTAT in Geneva, Switzerland and IMS conference in Sydney. Having exhausted local funding, she has now convinced several other organizations to support her conference habit.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/ZzhaVCrH8LU" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/travelling-thilaksha/Feeling the FPP love
http://feedproxy.google.com/~r/Hyndsight/~3/zijxxlxrCL8/
Fri, 10 Apr 2015 04:43:14 +0000https://robjhyndman.com/hyndsight/fpp-reviews/It is now exactly 12 months since the print version of my forecasting textbook with George Athanasopoulos was released on Amazon.com. Although the book is freely available online, it seems that a lot of people still like to buy print books.It’s nice to see that it has been getting some good reviews. It is rated 4.6 stars on Amazon.com with 6 out of 8 reviewers giving it 5 stars (the 3 reviewers on Amazon.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/zijxxlxrCL8" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/fpp-reviews/Paperpile makes me more productive
http://feedproxy.google.com/~r/Hyndsight/~3/x-5XX-now6I/
Thu, 09 Apr 2015 02:42:40 +0000https://robjhyndman.com/hyndsight/paperpile/One of the first things I tell my new research students is to use a reference management system to help them keep track of the papers they read, and to assist in creating bib files for their bibliography. Most of them use Mendeley, one or two use Zotero. Both do a good job and both are free.
I use neither. I did use Mendeley for several years, but it became slower and slower to sync as my reference collection grew.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/x-5XX-now6I" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/paperpile/Help
http://feedproxy.google.com/~r/Hyndsight/~3/RuUlcS_8GJA/
Wed, 08 Apr 2015 22:13:31 +0000https://robjhyndman.com/hyndsight/help/This is not a help service for all your R and forecasting questions, so please don’t post questions in the comments, or send them to me by email.
If you have questions about data analysis, ask for help on crossvalidated.com.
If you have questions about R, ask for help on stackoverflow.com.
If you think you have found a bug in one of my R packages, report it on github as explained here.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/RuUlcS_8GJA" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/help/A new open source data set for detecting time series outliers
http://feedproxy.google.com/~r/Hyndsight/~3/kIm_RYFDE6Q/
Wed, 01 Apr 2015 02:33:44 +0000https://robjhyndman.com/hyndsight/yahoo-data/Yahoo Labs has just released an interesting new data set useful for research on detecting anomalies (or outliers) in time series data. There are many contexts in which anomaly detection is important. For Yahoo, the main use case is in detecting unusual traffic on Yahoo servers.The data set comprises real traffic to Yahoo services, along with some synthetic data. There are 367 time series in the data set, each of which contains between 741 and 1680 observations recorded at regular intervals.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/kIm_RYFDE6Q" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/yahoo-data/What to cite?
http://feedproxy.google.com/~r/Hyndsight/~3/U43fvmoklCA/
Wed, 25 Mar 2015 03:27:57 +0000https://robjhyndman.com/hyndsight/what-to-cite/This question comes from a comment on another post:
I’ve seen authors citing as many references as possible to try to please potential referees. Many of those references are low quality papers though. Any general guidance about a typical length for the reference section?
It depends on the subject and style of the paper. I’ve written a paper with over 900 citations, but that was a review of time series forecasting over a 25 year period, and so it had to include a lot of references.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/U43fvmoklCA" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/what-to-cite/New job opportunities at Monash
http://feedproxy.google.com/~r/Hyndsight/~3/jLG1Cxs4NZE/
Tue, 24 Mar 2015 05:21:08 +0000https://robjhyndman.com/hyndsight/maxima-jobs/We are now advertising for various positions in applied statistics, operations research and applied mathematics.
Click here for details These jobs are with MAXIMA (the Monash Academy for Cross & Interdisciplinary Mathematical Applications).
Please do not send any questions to me (I won’t answer). Click above and follow the instructions.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/jLG1Cxs4NZE" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/maxima-jobs/Two new interviews
http://feedproxy.google.com/~r/Hyndsight/~3/OJKY937Ucig/
Fri, 20 Mar 2015 12:20:42 +0000https://robjhyndman.com/hyndsight/mbs-interviews/I was recently interviewed as part of a promotion for the Monash Business School. The interviews can be watched below if anyone is interested. The titles chosen weren’t my ideas.Predicting the future: pushing the boundaries of electricity forecasting How can you use data science to future-proof your business?<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/OJKY937Ucig" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/mbs-interviews/Dark themes for writing
http://feedproxy.google.com/~r/Hyndsight/~3/ebrNXeU60bk/
Wed, 18 Mar 2015 04:55:11 +0000https://robjhyndman.com/hyndsight/dark-themes-for-writing/I spend much of my day sitting in front of a screen, coding or writing. To limit the strain on my eyes, I use a dark theme as much as possible. That is, I write with light colored text on a dark background. I don’t know why this is not the default in more software as it makes a big difference after a few hours of writing.
Most of the time, I am writing using either Sublime Text, RStudio or TeXstudio.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/ebrNXeU60bk" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/dark-themes-for-writing/Common reasons for rejection
http://feedproxy.google.com/~r/Hyndsight/~3/LFZm1VOoYBg/
Thu, 12 Mar 2015 04:45:30 +0000https://robjhyndman.com/hyndsight/ijf-rejections/Every week I reject some papers submitted to the International Journal of Forecasting, without sending the papers off to associate editors or reviewers. Here are five of the most common reasons for rejection.1. Wrong Journal Submissions to the IJF should be about forecasting, obviously. But we often get papers on econometrics, or time series analysis, or something else that is not forecasting. Even if a paper has some implications for forecasting, if these are not discussed at all, the paper is not within scope for the journal.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/LFZm1VOoYBg" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/ijf-rejections/Statistical modelling and analysis of big data
http://feedproxy.google.com/~r/Hyndsight/~3/KZzGmWS3DTc/
Sun, 22 Feb 2015 23:00:54 +0000https://robjhyndman.com/hyndsight/bigdata2015/I’m currently attending the one day workshop on this topic at QUT in Brisbane. This morning I spoke on “Visualizing and forecasting big time series data”. My slides are here.
The talks are being streamed.
OVERVIEW Big data is now endemic in business, industry, government, environmental management, medical science, social research and so on. One of the commensurate challenges is how to effectively model and analyse these data.
This workshop will bring together national and international experts in statistical modelling and analysis of big data, to share their experiences, approaches and opinions about future directions in this field.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/KZzGmWS3DTc" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/bigdata2015/Thanks Paul and welcome Dilek
http://feedproxy.google.com/~r/Hyndsight/~3/L5U7jSU9kG4/
Mon, 09 Feb 2015 22:45:34 +0000https://robjhyndman.com/hyndsight/ijf-change-of-editors/Today, there is a change in editors at the International Journal of Forecasting. Paul Goodwin is retiring from the editorial board, and Dilek Önkal is taking his place.Paul Goodwin was appointed as an associate editor in 1999 and as an editor in 2010. Paul is retiring from his position as Professor of Management Science at the University of Bath, UK, and has decided to also retire from the IJF editorial board.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/L5U7jSU9kG4" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/ijf-change-of-editors/Standard error: a poem
http://feedproxy.google.com/~r/Hyndsight/~3/laDpt-YUI2Q/
Wed, 04 Feb 2015 06:43:19 +0000https://robjhyndman.com/hyndsight/standard-error-poem/This poem was written by David Goddard from the Monash University Department of Epidemiology and Preventive Medicine. It is reproduced here with his permission. The poem won the inaugural Monash University poetry competition and will soon be published in an anthology of contemporary poetry.For those who like this sort of thing (as I do), there is a nice collection of statistical poetry here.
Standard error David Gordon Goddard<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/laDpt-YUI2Q" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/standard-error-poem/IASC Data Analysis Competition 2015
http://feedproxy.google.com/~r/Hyndsight/~3/UZ1R8V5Z1no/
Mon, 02 Feb 2015 10:01:26 +0000https://robjhyndman.com/hyndsight/iasc-competition-2015/The International Association for Statistical Computing (IASC) is holding a Data Analysis Competition. Winners will be invited to present their work at the Joint Meeting of IASC-ABE Satellite Conference for the 60th ISI WSC 2015 to be held at Atlântico Búzios Convention & Resort in Búzios, RJ, Brazil (August 2-4, 2015). They will also be invited to submit a manuscript for possible publication (following peer review) to IASC’s official journal, Computational Statistics & Data Analysis.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/UZ1R8V5Z1no" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/iasc-competition-2015/RSS feeds for statistics and related journals
http://feedproxy.google.com/~r/Hyndsight/~3/PBKxi1R6U68/
Fri, 23 Jan 2015 23:48:49 +0000https://robjhyndman.com/hyndsight/rss-feeds/I’ve now resurrected the collection of research journals that I follow, and set it up as a shared collection in feedly. So anyone can easily subscribe to all of the same journals, or select a subset of them, to follow on feedly.There are about 90 journals on the list, mostly in statistics, but some from machine learning, operations research and econometrics. I excluded probability journals, and areas of application that are well outside my research interests (such as bioinformatics, psychology and pharmacology).<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/PBKxi1R6U68" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/rss-feeds/Seminars in Taiwan
http://feedproxy.google.com/~r/Hyndsight/~3/ejFOl2t72_I/
Mon, 05 Jan 2015 09:33:04 +0000https://robjhyndman.com/hyndsight/seminars-in-taiwan/I’m currently visiting Taiwan and I’m giving two seminars while I’m here — one at the National Tsing Hua University in Hsinchu, and the other at Academia Sinica in Taipei. Details are below for those who might be nearby.Automatic Time Series Forecasting College of Technology Management, Institute of Service Science, National Tsing Hua University, Hsinchu 時間及地點：2015.1.7 (Wed.) 5pm @ TSMC building, 6F, room 622. 台積館6F孫運璿紀念中心 Many applications require a large number of time series to be forecast completely automatically.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/ejFOl2t72_I" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/seminars-in-taiwan/Di Cook is moving to Monash
http://feedproxy.google.com/~r/Hyndsight/~3/EXc9td4pvBs/
Wed, 24 Dec 2014 05:29:42 +0000https://robjhyndman.com/hyndsight/di-cook-is-moving-to-monash/I’m delighted that Professor Dianne Cook will be joining Monash University in July 2015 as a Professor of Business Analytics. Di is an Australian who has worked in the US for the past 25 years, mostly at Iowa State University. She is moving back to Australia and joining the Department of Econometrics and Business Statistics in the Monash Business School, as part of our initiative in Business Analytics.
Di is a world leader in data visualization, and is well-known for her work on interactive graphics.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/EXc9td4pvBs" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/di-cook-is-moving-to-monash/New R package for electricity forecasting
http://feedproxy.google.com/~r/Hyndsight/~3/gD9gpc_OtTg/
Wed, 17 Dec 2014 06:24:53 +0000https://robjhyndman.com/hyndsight/mefm/Shu Fan and I have developed a model for electricity demand forecasting that is now widely used in Australia for long-term forecasting of peak electricity demand. It has become known as the “Monash Electricity Forecasting Model”. We have decided to release an R package that implements our model so that other people can easily use it. The package is called “MEFM” and is available on github. We will probably also put in on CRAN eventually.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/gD9gpc_OtTg" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/mefm/A time series classification contest
http://feedproxy.google.com/~r/Hyndsight/~3/qa44mOSkz_A/
Sun, 14 Dec 2014 22:17:16 +0000https://robjhyndman.com/hyndsight/ts-classification-contest/Amongst today’s email was one from someone running a private competition to classify time series. Here are the essential details.
The data are measurements from a medical diagnostic machine which takes 1 measurement every second, and after 32-1000 seconds, the time series must be classified into one of two classes. Some pre-classified training data is provided. It is not necessary to classify all the test data, but you do need to have relatively high accuracy on what is classified.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/qa44mOSkz_A" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/ts-classification-contest/Am I a data scientist?
http://feedproxy.google.com/~r/Hyndsight/~3/dpb8NLldgGI/
Tue, 09 Dec 2014 00:06:52 +0000https://robjhyndman.com/hyndsight/am-i-a-data-scientist/Last night I gave a very short talk (less than 5 minutes) at the Melbourne Analytics Charity Christmas Gala, a combined event of the Statistical Society of Australia, Data Science Melbourne, Big Data Analytics and Melbourne Users of R Network.
This is (roughly) what I said.Statisticians seem to go through regular periods of existential crisis as they worry about other groups of people who do data analysis. A common theme is: all these other people (usually computer scientists) are doing our job!<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/dpb8NLldgGI" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/am-i-a-data-scientist/Honoring Herman Stekler
http://feedproxy.google.com/~r/Hyndsight/~3/TQ6deqz97Is/
Mon, 08 Dec 2014 00:11:59 +0000https://robjhyndman.com/hyndsight/honoring-herman-stekler/The first issue of the IJF for 2015 has just been published, and I’m delighted that it includes a special section honoring Herman Stekler. It includes articles covering a range of his forecasting interests, although not all of them (sports forecasting is missing). Herman himself wrote a paper for it looking at “Forecasting—Yesterday, Today and Tomorrow”.
He is in a unique position to write such a paper as he has been doing forecasting research longer than anyone else on the planet — his first published paper on forecasting appeared in 1959.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/TQ6deqz97Is" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/honoring-herman-stekler/Prediction competitions
http://feedproxy.google.com/~r/Hyndsight/~3/fm5f1u3OTLo/
Fri, 05 Dec 2014 12:05:33 +0000https://robjhyndman.com/hyndsight/prediction-competitions/Competitions have a long history in forecasting and prediction, and have been instrumental in forcing research attention on methods that work well in practice. In the forecasting community, the M competition and M3 competition have been particularly influential. The data mining community have the annual KDD cup which has generated attention on a wide range of prediction problems and associated methods. Recent KDD cups are hosted on kaggle.
In my research group meeting today, we discussed our (limited) experiences in competing in some Kaggle competitions, and we reviewed the following two papers which describe two prediction competitions:<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/fm5f1u3OTLo" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/prediction-competitions/New Australian data on the HMD
http://feedproxy.google.com/~r/Hyndsight/~3/-Vt2-v9i8Y4/
Thu, 27 Nov 2014 02:33:46 +0000https://robjhyndman.com/hyndsight/hmd-australia/The Human Mortality Database is a wonderful resource for anyone interested in demographic data. It is a carefully curated collection of high quality deaths and population data from 37 countries, all in a consistent format with consistent definitions. I have used it many times and never cease to be amazed at the care taken to maintain such a great resource.
The data are continually being revised and updated. Today the Australian data has been updated to 2011.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/-Vt2-v9i8Y4" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/hmd-australia/Visualization of probabilistic forecasts
http://feedproxy.google.com/~r/Hyndsight/~3/d7yYnaC-IRc/
Fri, 21 Nov 2014 06:40:51 +0000https://robjhyndman.com/hyndsight/visualization-of-probabilistic-forecasts/This week my research group discussed Adrian Raftery’s recent paper on “Use and Communication of Probabilistic Forecasts” which provides a fascinating but brief survey of some of his work on modelling and communicating uncertain futures. Coincidentally, today I was also sent a copy of David Spiegelhalter’s paper on “Visualizing Uncertainty About the Future”. Both are well-worth reading.
It made me think about my own efforts to communicate future uncertainty through graphics.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/d7yYnaC-IRc" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/visualization-of-probabilistic-forecasts/IJF review papers
http://feedproxy.google.com/~r/Hyndsight/~3/6ql8MQVUPtw/
Tue, 11 Nov 2014 23:50:32 +0000https://robjhyndman.com/hyndsight/ijf-review-papers/Review papers are extremely useful for new researchers such as PhD students, or when you want to learn about a new research field. The International Journal of Forecasting produced a whole review issue in 2006, and it contains some of the most highly cited papers we have ever published. Now, beginning with the latest issue of the journal, we have started publishing occasional review articles on selected areas of forecasting. The first two articles are:<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/6ql8MQVUPtw" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/ijf-review-papers/Seasonal periods
http://feedproxy.google.com/~r/Hyndsight/~3/TMnHFHuQvQU/
Fri, 07 Nov 2014 01:43:08 +0000https://robjhyndman.com/hyndsight/seasonal-periods/I get questions about this almost every week. Here is an example from a recent comment on this blog:
I have two large time series data. One is separated by seconds intervals and the other by minutes. The length of each time series is 180 days. I’m using R (3.1.1) for forecasting the data. I’d like to know the value of the “frequency” argument in the ts() function in R, for each data set.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/TMnHFHuQvQU" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/seasonal-periods/ABS seasonal adjustment update
http://feedproxy.google.com/~r/Hyndsight/~3/eHnztqdi-nQ/
Wed, 05 Nov 2014 04:56:58 +0000https://robjhyndman.com/hyndsight/abs-seasonal-adjustment-3/Since my last post on the seasonal adjustment problems at the Australian Bureau of Statistics, I’ve been working closely with people within the ABS to help them resolve the problems in time for tomorrow’s release of the October unemployment figures.
Now that the ABS has put out a statement about the problem, I thought it would be useful to explain the underlying methodology for those who are interested.The Labour Force Survey The unemployment rate is derived from the monthly Labour Force Survey.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/eHnztqdi-nQ" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/abs-seasonal-adjustment-3/Jobs at Amazon
http://feedproxy.google.com/~r/Hyndsight/~3/N2433br8W8s/
Fri, 31 Oct 2014 02:13:54 +0000https://robjhyndman.com/hyndsight/jobs-at-amazon/I do not normally post job adverts, but this was very specifically targeted to “applied time series candidates” so I thought it might be of sufficient interest to readers of this blog.Here is an excerpt from an email I received from someone at Amazon:
Amazon is aggressively recruiting in the data sciences, and we have found that applied economists compare quite favorably with the machine learning specialists and statisticians that are sometimes recruited for such roles.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/N2433br8W8s" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/jobs-at-amazon/Prediction intervals too narrow
http://feedproxy.google.com/~r/Hyndsight/~3/BWQJaJJBzyY/
Wed, 22 Oct 2014 05:25:12 +0000https://robjhyndman.com/hyndsight/narrow-pi/Almost all prediction intervals from time series models are too narrow. This is a well-known phenomenon and arises because they do not account for all sources of uncertainty. In my 2002 IJF paper, we measured the size of the problem by computing the actual coverage percentage of the prediction intervals on hold-out samples. We found that for ETS models, nominal 95% intervals may only provide coverage between 71% and 87%. The difference is due to missing sources of uncertainty.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/BWQJaJJBzyY" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/narrow-pi/hts with regressors
http://feedproxy.google.com/~r/Hyndsight/~3/hdBLbGKEiVw/
Mon, 20 Oct 2014 02:08:07 +0000https://robjhyndman.com/hyndsight/hts-with-regressors/The hts package for R allows for forecasting hierarchical and grouped time series data. The idea is to generate forecasts for all series at all levels of aggregation without imposing the aggregation constraints, and then to reconcile the forecasts so they satisfy the aggregation constraints. (An introduction to reconciling hierarchical and grouped time series is available in this Foresight paper.)
The base forecasts can be generated using any method, with ETS models and ARIMA models provided as options in the forecast.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/hdBLbGKEiVw" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/hts-with-regressors/Congratulations to Dr Souhaib Ben Taieb
http://feedproxy.google.com/~r/Hyndsight/~3/ENG655BmLOY/
Wed, 15 Oct 2014 01:38:10 +0000https://robjhyndman.com/hyndsight/souhaib/Souhaib Ben Taieb has been awarded his doctorate at the Université libre de Bruxelles and so he is now officially Dr Ben Taieb! Although Souhaib lives in Brussels, and was a student at the Université libre de Bruxelles, I co-supervised his doctorate (along with Professor Gianluca Bontempi). Souhaib is the 19th PhD student of mine to graduate.
His thesis was on “Machine learning strategies for multi-step-ahead time series forecasting” and is now available online.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/ENG655BmLOY" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/souhaib/Explaining the ABS unemployment fluctuations
http://feedproxy.google.com/~r/Hyndsight/~3/H45dWGpSgEQ/
Fri, 10 Oct 2014 07:06:04 +0000https://robjhyndman.com/hyndsight/abs-seasonal-adjustment-2/Although the Guardian claimed yesterday that I had explained “what went wrong” in the July and August unemployment figures, I made no attempt to do so as I had no information about the problems. Instead, I just explained a little about the purpose of seasonal adjustment.
However, today I learned a little more about the ABS unemployment data problems, including what may be the explanation for the fluctuations. This explanation was offered by Westpac’s chief economist, Bill Evans (see here for a video of him explaining the issue).<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/H45dWGpSgEQ" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/abs-seasonal-adjustment-2/Seasonal adjustment in the news
http://feedproxy.google.com/~r/Hyndsight/~3/Ttx8qTYIH94/
Thu, 09 Oct 2014 05:50:51 +0000https://robjhyndman.com/hyndsight/abs-seasonal-adjustment/It’s not every day that seasonal adjustment makes the front page of the newspapers, but it has today with the ABS saying that the recent seasonally adjusted unemployment data would be revised.
I was interviewed about the underlying concepts for the Guardian in this piece.
Further comment from me about users paying for the ABS data is here.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Ttx8qTYIH94" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/abs-seasonal-adjustment/Connect with local employers
http://feedproxy.google.com/~r/Hyndsight/~3/eLYcfslG5-0/
Tue, 07 Oct 2014 23:07:25 +0000https://robjhyndman.com/hyndsight/connect-with-local-employers/I keep telling students that there are lots of jobs in data science (including statistics), and they often tell me they can’t find them advertised. As usual, you do have to do some networking, and one of the best ways of doing it is via a Data Science Meetup. Many cities now have them including Melbourne, Sydney, London, etc. It is the perfect opportunity to meet with local employers, many of which are hiring due to the huge expansion in the use of data analysis in business (aka business analytics).<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/eLYcfslG5-0" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/connect-with-local-employers/IIF Sponsored Workshops
http://feedproxy.google.com/~r/Hyndsight/~3/twh5qq6nszQ/
Mon, 06 Oct 2014 22:57:53 +0000https://robjhyndman.com/hyndsight/iif-workshops/The International Institute of Forecasters sponsors workshops every year, each of which focuses on a specific theme. The purpose of these workshops is to facilitate small, informal meetings where experts in a particular field of forecasting can discuss forecasting problems, research, and solutions. Over the years, our workshops have covered topics from Predicting Rare Events, ICT Forecasting, and, most recently, Singular Spectrum Analysis. Often these workshops are associated with a special issue of the International Journal of Forecasting.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/twh5qq6nszQ" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/iif-workshops/TBATS with regressors
http://feedproxy.google.com/~r/Hyndsight/~3/MEPokX0WMP0/
Mon, 06 Oct 2014 00:57:49 +0000https://robjhyndman.com/hyndsight/tbats-with-regressors/I’ve received a few emails about including regression variables (i.e., covariates) in TBATS models. As TBATS models are related to ETS models, tbats() is unlikely to ever include covariates as explained here. It won’t actually complain if you include an xreg argument, but it will ignore it.
When I want to include covariates in a time series model, I tend to use auto.arima() with covariates included via the xreg argument. If the time series has multiple seasonal periods, I use Fourier terms as additional covariates.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/MEPokX0WMP0" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/tbats-with-regressors/FPP now available as a downloadable e-book
http://feedproxy.google.com/~r/Hyndsight/~3/SGI5liSV9dk/
Sun, 21 Sep 2014 01:25:11 +0000https://robjhyndman.com/hyndsight/fpp-e-book/My forecasting textbook with George Athanasopoulos is already available online (for free), and in print via Amazon (for under $40). Now we have made it available as a downloadable e-book via Google Books (for $15.55). The Google Books version is identical to the print version on Amazon (apart from a few typos that have been fixed).
To use the e-book version on an iPad or Android tablet, you need to have the Google Books app installed [iPad, Android].<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/SGI5liSV9dk" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/fpp-e-book/Tim Harford on forecasting
http://feedproxy.google.com/~r/Hyndsight/~3/prGNFqw_yi4/
Mon, 08 Sep 2014 10:15:40 +0000https://robjhyndman.com/hyndsight/tim-harford-on-forecasting/A few weeks ago I had a Skype chat with Tim Harford, the “Undercover Economist” for Britain’s Financial Times. He was working on an article for the FT on forecasting, and wanted my perspective as an academic forecaster. I mostly talked about what makes some things more predictable than others, as discussed in this blog post. In the end, his article headed in a different direction, so I don’t get quoted, but it is still a good read!<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/prGNFqw_yi4" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/tim-harford-on-forecasting/Generating quantile forecasts in R
http://feedproxy.google.com/~r/Hyndsight/~3/UHzonfy9QdM/
Mon, 08 Sep 2014 05:36:56 +0000https://robjhyndman.com/hyndsight/quantile-forecasts-in-r/From today’s email:
I have just finished reading a copy of ‘Forecasting:Principles and Practice’ and I have found the book really interesting. I have particularly enjoyed the case studies and focus on practical applications.
After finishing the book I have joined a forecasting competition to put what I’ve learnt to the test. I do have a couple of queries about the forecasting outputs required. The output required is a quantile forecast, is this the same as prediction intervals?<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/UHzonfy9QdM" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/quantile-forecasts-in-r/Resources for the FPP book
http://feedproxy.google.com/~r/Hyndsight/~3/y3CCan0Vpvk/
Wed, 03 Sep 2014 01:59:28 +0000https://robjhyndman.com/hyndsight/fpp-resources/The FPP resources page has recently been updated with several new additions including
R code for all examples in the book. This was already available within each chapter, but the examples have been collected into one file per chapter to save copying and pasting the various code fragments.
Slides from a course on Predictive Analytics from the University of Sydney.
Slides from a course on Economic Forecasting from the University of Hawaii.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/y3CCan0Vpvk" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/fpp-resources/A new candidate for worst figure
http://feedproxy.google.com/~r/Hyndsight/~3/gNrVmGiLIAQ/
Mon, 01 Sep 2014 05:03:22 +0000https://robjhyndman.com/hyndsight/worst-figure/Today I read a paper that had been submitted to the IJF which included the following figure
along with several similar plots. (Click for a larger version.) I haven’t seen anything this bad for a long time. In fact, I think I would find it very difficult to reproduce using R, or even Excel (which is particularly adept at bad graphics).
A few years ago I produced “Twenty rules for good graphics”.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/gNrVmGiLIAQ" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/worst-figure/Forecasting with R in WA
http://feedproxy.google.com/~r/Hyndsight/~3/CUonL2KNAgk/
Sun, 24 Aug 2014 22:53:49 +0000https://robjhyndman.com/hyndsight/forecasting-with-r-in-wa/On 23-25 September, I will be running a 3-day workshop in Perth on “Forecasting: principles and practice” mostly based on my book of the same name.
Workshop participants will be assumed to be familiar with basic statistical tools such as multiple regression, but no knowledge of time series or forecasting will be assumed. Some prior experience in R is highly desirable.
Venue: The University Club, University of Western Australia, Nedlands WA.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/CUonL2KNAgk" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/forecasting-with-r-in-wa/biblatex for statisticians
http://feedproxy.google.com/~r/Hyndsight/~3/HmQrAFl1v6Q/
Fri, 22 Aug 2014 03:37:16 +0000https://robjhyndman.com/hyndsight/biblatex-for-statisticians/I am now using biblatex for all my bibliographic work as it seems to have developed enough to be stable and reliable. The big advantage of biblatex is that it is easy to format the bibliography to conform to specific journal or publisher styles. It is also possible to have structured bibliographies (e.g., divided into sections: books, papers, R packages, etc.) Here is my default setting which should be suitable for almost all statistics and econometrics journals.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/HmQrAFl1v6Q" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/biblatex-for-statisticians/GEFCom 2014 energy forecasting competition is underway
http://feedproxy.google.com/~r/Hyndsight/~3/z1-JnYWU9jk/
Mon, 18 Aug 2014 05:44:07 +0000https://robjhyndman.com/hyndsight/gefcom-2014/GEFCom 2014 is the most advanced energy forecasting competition ever organized, both in terms of the data involved, and in terms of the way the forecasts will be evaluated.
So everyone interested in energy forecasting should head over to the competition webpage and start forecasting: www.gefcom.org.
This time, the competition is hosted on CrowdANALYTIX rather than Kaggle.
Highlights of GEFCom2014:
An upgraded edition from GEFCom2012
Four tracks: electric load, electricity price, wind power and solar power forecasting.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/z1-JnYWU9jk" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/gefcom-2014/Visit of Di Cook
http://feedproxy.google.com/~r/Hyndsight/~3/uE-W81OjxKk/
Tue, 12 Aug 2014 23:28:13 +0000https://robjhyndman.com/hyndsight/visit-of-di-cook/Next week, Professor Di Cook from Iowa State University is visiting my research group at Monash University. Di is a world leader in data visualization, and is especially well-known for her work on interactive graphics and the XGobi and GGobi software. See her book with Deb Swayne for details.
For those wanting to hear her speak, read on.Research seminar She will be giving a seminar at 2pm on Monday 18 August at the Monash Clayton campus (Rm E457, Menzies Building 11).<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/uE-W81OjxKk" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/visit-of-di-cook/What not to say in a job interview
http://feedproxy.google.com/~r/Hyndsight/~3/eR4TJSIOTKU/
Tue, 12 Aug 2014 11:48:33 +0000https://robjhyndman.com/hyndsight/what-not-to-say-in-a-job-interview/I’ve interviewed a few people for jobs at Monash University, and there’s always someone who comes out with something surprising. Here are some real examples.For a post-doctoral research position:
Q: What would you say were your major weaknesses?
A: I don’t have any.
Q: Really? You can’t think of anything that you could work on, new skills you could develop, anything at all that you might be able to improve?<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/eR4TJSIOTKU" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/what-not-to-say-in-a-job-interview/Minimal reproducible examples
http://feedproxy.google.com/~r/Hyndsight/~3/mpoO-fMmgNI/
Mon, 11 Aug 2014 04:54:39 +0000https://robjhyndman.com/hyndsight/minimal-reproducible-examples/I occasionally get emails from people thinking they have found a bug in one of my R packages, and I usually have to reply asking them to provide a minimal reproducible example (MRE). This post is to provide instructions on how to create a MRE.Bug reports on github, not email First, if you think there is a bug, please don’t send me emails. Instead, use the bug-reporting facility on github.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/mpoO-fMmgNI" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/minimal-reproducible-examples/Student forecasting awards from the IIF
http://feedproxy.google.com/~r/Hyndsight/~3/kh5YaJYZowg/
Sat, 26 Jul 2014 03:14:48 +0000https://robjhyndman.com/hyndsight/iif-awards/At the IIF annual board meeting last month in Rotterdam, I suggested that we provide awards to the top students studying forecasting at university level around the world, to the tune of $100 plus IIF membership for a year. I’m delighted that the idea met with enthusiasm, and that the awards are now available. Even better, my second year forecasting subject has been approved for an award.
The IIF have agreed to fund awards for 20 forecasting courses to start with.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/kh5YaJYZowg" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/iif-awards/Coherent population forecasting using R
http://feedproxy.google.com/~r/Hyndsight/~3/okXJUAo5FF4/
Thu, 24 Jul 2014 02:03:01 +0000https://robjhyndman.com/hyndsight/coherent-population-forecasting/This is an example of how to use the demography package in R for stochastic population forecasting with coherent components. It is based on the papers by Hyndman and Booth (IJF 2008) and Hyndman, Booth and Yasmeen (Demography 2013). I will use Australian data from 1950 to 2009 and forecast the next 50 years.
In demography, “coherent” forecasts are where male and females (or other sub-groups) do not diverge over time.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/okXJUAo5FF4" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/coherent-population-forecasting/Plotting the characteristic roots for ARIMA models
http://feedproxy.google.com/~r/Hyndsight/~3/wpV68QH6QYU/
Wed, 23 Jul 2014 07:57:28 +0000https://robjhyndman.com/hyndsight/arma-roots/When modelling data with ARIMA models, it is sometimes useful to plot the inverse characteristic roots. The following functions will compute and plot the inverse roots for any fitted ARIMA model (including seasonal models). # Compute AR roots arroots <- function(object) { if(!("Arima" %in% class(object)) & !("ar" %in% class(object))) stop("object must be of class Arima or ar") if("Arima" %in% class(object)) parvec <- object$model$phi else parvec <- object$ar if(length(parvec) > 0) { last.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/wpV68QH6QYU" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/arma-roots/I am not an econometrician
http://feedproxy.google.com/~r/Hyndsight/~3/YhZvsPjjYhc/
Mon, 21 Jul 2014 03:24:09 +0000https://robjhyndman.com/hyndsight/statistics-vs-econometrics/I am a statistician, but I have worked in a department of predominantly econometricians for the past 17 years. It is a little like an Australian visiting the United States. Initially, it seems that we talk the same language, do the same sorts of things, and have a very similar culture. But the longer you stay there, the more you realise there are differences that run deep and affect the way you see the world.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/YhZvsPjjYhc" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/statistics-vs-econometrics/Variations on rolling forecasts
http://feedproxy.google.com/~r/Hyndsight/~3/YGvrX1WN9So/
Tue, 15 Jul 2014 23:49:00 +0000https://robjhyndman.com/hyndsight/rolling-forecasts/Rolling forecasts are commonly used to compare time series models. Here are a few of the ways they can be computed using R. I will use ARIMA models as a vehicle of illustration, but the code can easily be adapted to other univariate time series models.One-step forecasts without re-estimation The simplest approach is to estimate the model on a single set of training data, and then compute one-step forecasts on the remaining test data.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/YGvrX1WN9So" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/rolling-forecasts/SAS/IIF grants
http://feedproxy.google.com/~r/Hyndsight/~3/Xlz58LZD8rs/
Mon, 14 Jul 2014 22:43:27 +0000https://robjhyndman.com/hyndsight/sasiif-grants/Every year, the International Institute of Forecasters in conjunction with SAS offer some small grants to help promote research in forecasting. There are two $5000 grants per year for research on forecasting methodology and applications. This year, applications close on 30 September 2014. More details are given here.
Information about past SAS-IIF awards is given on the IIF website. It is interesting to see the range of topics covered. Here are the winning projects in the last two years:<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Xlz58LZD8rs" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/sasiif-grants/Varian on big data
http://feedproxy.google.com/~r/Hyndsight/~3/Suler_pbySY/
Sun, 15 Jun 2014 22:45:21 +0000https://robjhyndman.com/hyndsight/varian-2014/Last week my research group discussed Hal Varian’s interesting new paper on “Big data: new tricks for econometrics”, Journal of Economic Perspectives, 28(2): 3-28.
It’s a nice introduction to trees, bagging and forests, plus a very brief entree to the LASSO and the elastic net, and to slab and spike regression. Not enough to be able to use them, but ok if you’ve no idea what they are. It was more disappointing on boosting (completely ignoring the fact that boosting can be applied in a regression context as well as a classification context), and his comments on causality seemed curiously naive.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Suler_pbySY" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/varian-2014/Specifying complicated groups of time series in hts
http://feedproxy.google.com/~r/Hyndsight/~3/fCJf8dk3in0/
Sun, 15 Jun 2014 03:26:38 +0000https://robjhyndman.com/hyndsight/gts/With the latest version of the hts package for R, it is now possible to specify rather complicated grouping structures relatively easily.
All aggregation structures can be represented as hierarchies or as cross-products of hierarchies. For example, a hierarchical time series may be based on geography: country, state, region, store. Often there is also a separate product hierarchy: product groups, product types, packet size. Forecasts of all the different types of aggregation are required; e.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/fCJf8dk3in0" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/gts/European talks. June-July 2014
http://feedproxy.google.com/~r/Hyndsight/~3/PR7MLIMsmMs/
Sat, 14 Jun 2014 06:44:09 +0000https://robjhyndman.com/hyndsight/europe2014/For the next month I am travelling in Europe and will be giving the following talks.
17 June. Challenges in forecasting peak electricity demand. Energy Forum, Sierre, Valais/Wallis, Switzerland.
20 June. Common functional principal component models for mortality forecasting. International Workshop on Functional and Operatorial Statistics. Stresa, Italy.
24-25 June. Functional time series with applications in demography. Humboldt University, Berlin.
1 July. Fast computation of reconciled forecasts in hierarchical and grouped time series.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/PR7MLIMsmMs" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/europe2014/Creating a handout from beamer slides
http://feedproxy.google.com/~r/Hyndsight/~3/Ax8XxGOxAus/
Wed, 11 Jun 2014 01:29:22 +0000https://robjhyndman.com/hyndsight/beamer-handout/I’m about to head off on a speaking tour to Europe (more on that in another post) and one of my hosts has asked for my powerpoint slides so they can print them. They have made two false assumptions: (1) that I use powerpoint; (2) that my slides are static so they can be printed.
Instead, I produced a cut-down version of my beamer slides, leaving out some of the animations and other features that will not print easily.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Ax8XxGOxAus" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/beamer-handout/Data science market places
http://feedproxy.google.com/~r/Hyndsight/~3/5Fa8pnRenbs/
Mon, 26 May 2014 02:42:44 +0000https://robjhyndman.com/hyndsight/marketplaces/Some new websites are being established offering “market places” for data science. Two I’ve come across recently are Experfy and SnapAnalytx.Experfy provides a way for companies to find statisticians and other data scientists, either for short-term consultancies, or to fill full-time positions. They describe their “providers” as “Data Engineers, Data Scientists, Data Mining Experts, Data Analyst/Modelers, Big Data Solutions Architects, Visualization Designers, Statisticians, Applied Physicists, Mathematicians, Econometricians and Bioinformaticians.” Data scientists can sign up as “Providers”, companies can sign up as “Clients”.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/5Fa8pnRenbs" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/marketplaces/Structural breaks
http://feedproxy.google.com/~r/Hyndsight/~3/bAWpBBgdWHY/
Fri, 23 May 2014 13:32:32 +0000https://robjhyndman.com/hyndsight/structural-breaks/I’m tired of reading about tests for structural breaks and here’s why.
A structural break occurs when we see a sudden change in a time series or a relationship between two time series. Econometricians love papers on structural breaks, and apparently believe in them. Personally, I tend to take a different view of the world. I think a more realistic view is that most things change slowly over time, and only occasionally with sudden discontinuous change.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/bAWpBBgdWHY" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/structural-breaks/To explain or predict?
http://feedproxy.google.com/~r/Hyndsight/~3/PWZBG1oWyQU/
Mon, 19 May 2014 05:35:33 +0000https://robjhyndman.com/hyndsight/to-explain-or-predict/Last week, my research group discussed Galit Shmueli’s paper “To explain or to predict?”, Statistical Science, 25(3), 289-310. (See her website for further materials.) This is a paper everyone doing statistics and econometrics should read as it helps to clarify a distinction that is often blurred. In the discussion, the following issues were covered amongst other things.
The AIC is better suited to model selection for prediction as it is asymptotically equivalent to leave-one-out cross-validation in regression, or one-step-cross-validation in time series.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/PWZBG1oWyQU" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/to-explain-or-predict/Questions on the business analytics jobs
http://feedproxy.google.com/~r/Hyndsight/~3/8uFvFMDgKOk/
Tue, 13 May 2014 02:27:33 +0000https://robjhyndman.com/hyndsight/business-analytics-jobs-questions/I’ve received a few questions on the business analytics jobs advertised last week. I think it is best if I answer them here so other potential candidates can have the same information. I will add to this post if I receive more questions.1. What are your expectations in terms of outputs (KPIs)? Typically, a person at Level B (Lecturer) in our department would be producing at least one refereed article in a good scholarly journal per year.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/8uFvFMDgKOk" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/business-analytics-jobs-questions/ARIMA models with long lags
http://feedproxy.google.com/~r/Hyndsight/~3/ifOLj640drM/
Thu, 08 May 2014 03:43:23 +0000https://robjhyndman.com/hyndsight/arima-models-with-long-lags/Today’s email question:
I work within a government budget office and sometimes have to forecast fairly simple time series several quarters into the future. Auto.arima() works great and I often get something along the lines of: ARIMA(0,0,1)(1,1,0)[12] with drift as the lowest AICc.
However, my boss (who does not use R) takes issue with low-order AR and MA because “you’re essentially using forecasted data to make your forecast.” His models include AR(10) MA(12)s etc.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/ifOLj640drM" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/arima-models-with-long-lags/New jobs in business analytics at Monash
http://feedproxy.google.com/~r/Hyndsight/~3/gdBwbtXxGHg/
Sun, 04 May 2014 12:05:19 +0000https://robjhyndman.com/hyndsight/monash-business-analytics/We have an exciting new initiative at Monash University with some new positions in business analytics. This is part of a plan to strengthen our research and teaching in the data science/computational statistics area. We are hoping to make multiple appointments, at junior and senior levels. These are five-year appointments, but we hope that the positions will continue after that if we can secure suitable funding.What is business analytics? You can think of “business analytics” as the application of data analysis to business problems.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/gdBwbtXxGHg" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/monash-business-analytics/Great papers to read
http://feedproxy.google.com/~r/Hyndsight/~3/x5MHhCO2gpk/
Fri, 02 May 2014 03:14:33 +0000https://robjhyndman.com/hyndsight/great-papers/My research group meets every two weeks. It is always fun to talk about general research issues and new tools and tips we have discovered. We also use some of the time to discuss a paper that I choose for them. Today we discussed Breiman’s classic (2001) two cultures paper — something every statistician should read, including the discussion.
I select papers that I want every member of research team to be familiar with.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/x5MHhCO2gpk" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/great-papers/Past, present, and future of statistical science
http://feedproxy.google.com/~r/Hyndsight/~3/0L-z-uT-dBU/
Mon, 28 Apr 2014 11:36:39 +0000https://robjhyndman.com/hyndsight/ppfss/This is the title of a wonderful new book that has just been released, courtesy of the Committee of Presidents of Statistical Societies.
It can be freely downloaded from the COPSS website or a hard copy can be purchased on Amazon (for only a little over 10c per page which is not bad compared to other statistics books).
The book consists of 52 chapters spanning 622 pages. The full table of contents below shows its scope and the list of authors (a veritable who’s who in statistics).<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/0L-z-uT-dBU" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/ppfss/Publishing an R package in the Journal of Statistical Software
http://feedproxy.google.com/~r/Hyndsight/~3/p2Hb7HOeKcs/
Thu, 24 Apr 2014 05:41:29 +0000https://robjhyndman.com/hyndsight/jss-rpackages/I’ve been an editor of JSS for the last few years, and as a result I tend to get email from people asking me about publishing papers describing R packages in JSS. So for all those wondering, here are some general comments.JSS prefers to publish papers about packages where the package is on CRAN and has been there long enough to have matured (i.e., obvious bugs ironed out and a few active users).<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/p2Hb7HOeKcs" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/jss-rpackages/Seven forecasting blogs
http://feedproxy.google.com/~r/Hyndsight/~3/VUwX_bhNNcs/
Tue, 22 Apr 2014 00:33:17 +0000https://robjhyndman.com/hyndsight/seven-forecasting-blogs/There are several other blogs on forecasting that readers might be interested in. Here are seven worth following:
No Hesitations by Francis Diebold (Professor of Economics, University of Pennsylvania). Diebold needs no introduction to forecasters. He primarily covers forecasting in economics and finance, but also xkcd cartoons, graphics, research issues, etc.
Econometrics Beat by Dave Giles. Dave is a professor of economics at the University of Victoria (Canada), formerly from my own department at Monash University (Australia), and a native New Zealander.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/VUwX_bhNNcs" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/seven-forecasting-blogs/Errors on percentage errors
http://feedproxy.google.com/~r/Hyndsight/~3/bYa1_Q5Sc4U/
Wed, 16 Apr 2014 12:11:06 +0000https://robjhyndman.com/hyndsight/smape/The MAPE (mean absolute percentage error) is a popular measure for forecast accuracy and is defined as $$ \text{MAPE} = 100\text{mean}(|y_t - \hat{y}_t|/|y_t|) $$ where $y_t$ denotes an observation and $\hat{y}_t$ denotes its forecast, and the mean is taken over $t$.
Armstrong (1985, p.348) was the first (to my knowledge) to point out the asymmetry of the MAPE saying that “it has a bias favoring estimates that are below the actual values”.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/bYa1_Q5Sc4U" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/smape/Generating tables in LaTeX
http://feedproxy.google.com/~r/Hyndsight/~3/Ihq7Takw4L4/
Mon, 14 Apr 2014 23:23:58 +0000https://robjhyndman.com/hyndsight/generating-tables-in-latex/Typing tables in LaTeX can get messy, but there are some good tools to simplify the process. One I discovered this week is tablesgenerator.com, a web-based tool for generating LaTeX tables. It also allows the table to saved in other formats including HTML and Markdown. The interface is simple, but it does most things. For complicated tables, some additional formatting may be necessary.
Similar functionality is available via plugins in Excel, OpenOffice and Libreoffice — useful if the data for the table is already stored in a spreadsheet.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Ihq7Takw4L4" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/generating-tables-in-latex/My forecasting book now on Amazon
http://feedproxy.google.com/~r/Hyndsight/~3/UlXR9knWK7o/
Wed, 09 Apr 2014 00:15:44 +0000https://robjhyndman.com/hyndsight/fpp-amazon/For all those people asking me how to obtain a print version of my book “Forecasting: principles and practice” with George Athanasopoulos, you now can.
Order on Amazon.com
Order on Amazon.co.uk
Order on Amazon.fr
The online book will continue to be freely available. The print version of the book is intended to help fund the development of the OTexts platform.
The price is US$45, £27 or €35.
Compare that to $195 for my previous forecasting textbook, $150 for Fildes and Ord, or $182 for Gonzalez-Rivera.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/UlXR9knWK7o" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/fpp-amazon/Job at Center for Open Science
http://feedproxy.google.com/~r/Hyndsight/~3/9eapC3nDCb8/
Mon, 07 Apr 2014 22:36:23 +0000https://robjhyndman.com/hyndsight/cos-job/This looks like an interesting job.
Dear Dr. Hyndman,
I write from the Center for Open Science, a non-profit organization based in Charlottesville, Virginia in the United States, which is dedicated to improving the alignment between scientific values and scientific practices. We are dedicated to open source and open science.
We are reaching out to you to find out if you know anyone who might be interested in our Statistical and Methodological Consultant position.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/9eapC3nDCb8" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/cos-job/Interpreting noise
http://feedproxy.google.com/~r/Hyndsight/~3/0XI63XxX20M/
Sun, 06 Apr 2014 00:04:36 +0000https://robjhyndman.com/hyndsight/interpreting-noise/When watching the TV news, or reading newspaper commentary, I am frequently amazed at the attempts people make to interpret random noise.
For example, the latest tiny fluctuation in the share price of a major company is attributed to the CEO being ill. When the exchange rate goes up, the TV finance commentator confidently announces that it is a reaction to Chinese building contracts. No one ever says “The unemployment rate has dropped by 0.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/0XI63XxX20M" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/interpreting-noise/Getting a LaTeX system set up
http://feedproxy.google.com/~r/Hyndsight/~3/v_Pg1ekKWgI/
Fri, 04 Apr 2014 02:00:38 +0000https://robjhyndman.com/hyndsight/latex-setup/Today I was teaching the honours students in econometrics and economics about LaTeX. Here are some brief instructions on how to set up a LaTeX system on different operating systems.MS-Windows Download and run the setup program for MikTeX. Choose the “basic” system. Download and run the installer program for TeXstudio. Then run TeXstudio and start typing.
Mac OS Download and install MacTeX. Then run TeXshop and start typing.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/v_Pg1ekKWgI" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/latex-setup/Cover of my forecasting textbook
http://feedproxy.google.com/~r/Hyndsight/~3/Wn1Da3SDDfc/
Tue, 18 Mar 2014 06:17:44 +0000https://robjhyndman.com/hyndsight/fpp-cover/We now have a cover for the print version of my forecasting book with George Athanasopoulos.
It should be on Amazon in a couple of weeks. The book is also freely available online.
The cover was produced by Scarlett Rugers who I can happily recommend to anyone wanting a book cover designed.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Wn1Da3SDDfc" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/fpp-cover/Fast computation of cross-validation in linear models
http://feedproxy.google.com/~r/Hyndsight/~3/UPdjO8GxRQE/
Mon, 17 Mar 2014 10:08:42 +0000https://robjhyndman.com/hyndsight/loocv-linear-models/The leave-one-out cross-validation statistic is given by
$$ \text{CV} = \frac{1}{N} \sum_{i=1}^N e_{[i]}^2, $$ where ${e_{[i]} = y_{i} - \hat{y}_{[i]}} $, the observations are given by $y_{1},\dots,y_{N}$, and $\hat{y}_{[i]}$ is the predicted value obtained when the model is estimated with the $i\text{th}$ case deleted. This is also sometimes known as the PRESS (Prediction Residual Sum of Squares) statistic.
It turns out that for linear models, we do not actually have to estimate the model $N$ times, once for each omitted case.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/UPdjO8GxRQE" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/loocv-linear-models/Probabilistic forecasting by Gneiting and Katzfuss (2014)
http://feedproxy.google.com/~r/Hyndsight/~3/G4FWeNRmPvc/
Fri, 14 Mar 2014 05:57:04 +0000https://robjhyndman.com/hyndsight/gneiting/The IJF is introducing occasional review papers on areas of forecasting. We did a whole issue in 2006 reviewing 25 years of research since the International Institute of Forecasters was established. Since then, there has been a lot of new work in application areas such as call center forecasting and electricity price forecasting. In addition, there are areas we did not cover in 2006 including new product forecasting and forecasting in finance.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/G4FWeNRmPvc" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/gneiting/Testing for trend in ARIMA models
http://feedproxy.google.com/~r/Hyndsight/~3/ZyANegioG0k/
Wed, 12 Mar 2014 22:24:29 +0000https://robjhyndman.com/hyndsight/arima-trends/Today’s email brought this one:
I was wondering if I could get your opinion on a particular problem that I have run into during the reviewing process of an article.
Basically, I have an analysis where I am looking at a couple of time-series and I wanted to know if, over time there was an upward trend in the series. Inspection of the raw data suggests there is, but we want some statistical evidence for this.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/ZyANegioG0k" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/arima-trends/Unit root tests and ARIMA models
http://feedproxy.google.com/~r/Hyndsight/~3/RRrWLTET82U/
Wed, 12 Mar 2014 11:22:45 +0000https://robjhyndman.com/hyndsight/unit-root-tests/An email I received today:
I have a small problem. I have a time series called x :
If I use the default values of auto.arima(x), the best model is an ARIMA(1,0,0)
However, I tried the function ndiffs(x, test=“adf”) and ndiffs(x, test=“kpss”) as the KPSS test seems to be the default value, and the number of difference is 0 for the kpss test (consistent with the results of auto.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/RRrWLTET82U" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/unit-root-tests/Using old versions of R packages
http://feedproxy.google.com/~r/Hyndsight/~3/mSvqRlFXVdU/
Mon, 10 Mar 2014 04:02:25 +0000https://robjhyndman.com/hyndsight/old-r-packages/I received this email yesterday:
I have been using your ‘forecast’ package for more than a year now. I was on R version 2.15 until last week, but I am having issues with lubridate package, hence decided to update R version to R 3.0.1. In our organization even getting an open source application require us to go through a whole lot of approval processes. I asked for R 3.0.1, before I get approval for 3.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/mSvqRlFXVdU" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/old-r-packages/IJF news
http://feedproxy.google.com/~r/Hyndsight/~3/9czZ4k6FErY/
Fri, 07 Mar 2014 03:38:56 +0000https://robjhyndman.com/hyndsight/ijf-news/This is a short piece I wrote for the next issue of the Oracle newsletter produced by the International Institute of Forecasters.Special section topics We continue to publish special sections on selected topics. Because of the change in the way regular papers are now handled, we do not publish whole special issues any more. Rather, each issue has regular papers at the front, and if there are any special sections they appear at the back.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/9czZ4k6FErY" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/ijf-news/Highlighting the web
http://feedproxy.google.com/~r/Hyndsight/~3/9hZY43bZjnQ/
Thu, 06 Mar 2014 01:29:06 +0000https://robjhyndman.com/hyndsight/highlighting-the-web/Users of my new online forecasting book have asked about having a facility for personal highlighting of selected sections, as students often do with print books. We have plans to make this a built-in part of the platform, but for now it is possible to do it using a simple browser extension. This approach allows any website to be highlighted, so is even more useful than if we only had the facility on OTexts.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/9hZY43bZjnQ" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/highlighting-the-web/Forecasting weekly data
http://feedproxy.google.com/~r/Hyndsight/~3/lMMk8UFPs6E/
Tue, 04 Mar 2014 23:57:02 +0000https://robjhyndman.com/hyndsight/forecasting-weekly-data/This is another situation where Fourier terms are useful for handling the seasonality. Not only is the seasonal period rather long, it is non-integer (averaging 365.25/7 = 52.18). So ARIMA and ETS models do not tend to give good results, even with a period of 52 as an approximation.
Regression with ARIMA errors The simplest approach is a regression with ARIMA errors. Here is an example using weekly data on US finished motor gasoline products supplied (in thousands of barrels per day) from February 1991 to May 2005.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/lMMk8UFPs6E" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/forecasting-weekly-data/Fitting models to short time series
http://feedproxy.google.com/~r/Hyndsight/~3/OJNSjQMx11o/
Tue, 04 Mar 2014 02:20:22 +0000https://robjhyndman.com/hyndsight/short-time-series/Following my post on fitting models to long time series, I thought I’d tackle the opposite problem, which is more common in business environments.
I often get asked how _few_ data points can be used to fit a time series model. As with almost all sample size questions, there is no easy answer. It depends on the number of model parameters to be estimated and the amount of randomness in the data.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/OJNSjQMx11o" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/short-time-series/Fitting models to long time series
http://feedproxy.google.com/~r/Hyndsight/~3/wXPaxmdhdmA/
Sat, 01 Mar 2014 06:08:49 +0000https://robjhyndman.com/hyndsight/long-time-series/I received this email today:
I recall you made this very insightful remark somewhere that, fitting a standard arima model with too much data, ie. a very long time series, is a bad idea.
Can you elaborate why?
I can see the issue with noise, which compounds the ML estimation as the series gets too long. But is there anything else?
I’m not sure where I made a comment about this, but it is true that ARIMA models don’t work well for very long time series.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/wXPaxmdhdmA" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/long-time-series/More time series data online
http://feedproxy.google.com/~r/Hyndsight/~3/vy8OaimTD_Y/
Thu, 27 Feb 2014 06:34:34 +0000https://robjhyndman.com/hyndsight/comp-engine/Earlier this week I had coffee with Ben Fulcher who told me about his online collection comprising about 30,000 time series, mostly medical series such as ECG measurements, meteorological series, birdsong, etc. There are some finance series, but not many other data from a business or economic context, although he does include my Time Series Data Library. In addition, he provides Matlab code to compute a large number of characteristics. Anyone wanting to test time series algorithms on a large collection of data should take a look.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/vy8OaimTD_Y" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/comp-engine/The forecast mean after back-transformation
http://feedproxy.google.com/~r/Hyndsight/~3/LjVjI6Ts-Q0/
Tue, 25 Feb 2014 03:45:52 +0000https://robjhyndman.com/hyndsight/backtransforming/Many functions in the forecast package for R will allow a Box-Cox transformation. The models are fitted to the transformed data and the forecasts and prediction intervals are back-transformed. This preserves the coverage of the prediction intervals, and the back-transformed point forecast can be considered the median of the forecast densities (assuming the forecast densities on the transformed scale are symmetric). For many purposes, this is acceptable, but occasionally the mean forecast is required.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/LjVjI6Ts-Q0" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/backtransforming/Statistical politicians
http://feedproxy.google.com/~r/Hyndsight/~3/rev8D9McMjM/
Sun, 23 Feb 2014 23:42:05 +0000https://robjhyndman.com/hyndsight/statistical-politicians/Last week we had the pleasure of Professor Stephen Pollock (University of Leicester) visiting our Department, best known in academic circles for his work on time series filtering (see his papers, and his excellent book). But he has another career as a member of the UK House of Lords (under the name Viscount Hanworth – he is a hereditary peer).
It made me wonder how many other politicians have PhDs (or equivalent) in statistics, or at least in mathematics.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/rev8D9McMjM" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/statistical-politicians/Forecasting within limits
http://feedproxy.google.com/~r/Hyndsight/~3/4jfjbPQZaUs/
Fri, 21 Feb 2014 10:36:14 +0000https://robjhyndman.com/hyndsight/forecasting-within-limits/It is common to want forecasts to be positive, or to require them to be within some specified range \([a,b]\). Both of these situations are relatively easy to handle using transformations.Positive forecasts To impose a positivity constraint, simply work on the log scale. With the forecast package in R, this can be handled by specifying the Box-Cox parameter \(\lambda=0\). For example, consider the real price of a dozen eggs (1900-1993; in cents):<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/4jfjbPQZaUs" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/forecasting-within-limits/Backcasting in R
http://feedproxy.google.com/~r/Hyndsight/~3/HuuAuDZBUWQ/
Thu, 20 Feb 2014 04:20:29 +0000https://robjhyndman.com/hyndsight/backcasting/Sometimes it is useful to “backcast” a time series — that is, forecast in reverse time. Although there are no in-built R functions to do this, it is very easy to implement. Suppose x is our time series and we want to backcast for \(h\) periods. Here is some code that should work for most univariate time series. The example is non-seasonal, but the code will also work with seasonal data.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/HuuAuDZBUWQ" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/backcasting/Global energy forecasting competitions
http://feedproxy.google.com/~r/Hyndsight/~3/kCwmF0NhSNg/
Wed, 19 Feb 2014 07:12:50 +0000https://robjhyndman.com/hyndsight/gefcom2014/The 2012 GEFcom competition was a great success with several new innovative forecasting methods introduced. These have been published in the IJF as follows: Hong, Pinson and Fan. Global Energy Forecasting Competition 2012
Charleton and Singleton. A refined parametric model for short term load forecasting
Lloyd. GEFCom2012 hierarchical load forecasting: Gradient boosting machines and Gaussian processes
Nedelec, Cugliari and Goude: GEFCom2012: Electric load forecasting and backcasting with semi-parametric models<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/kCwmF0NhSNg" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/gefcom2014/Hierarchical forecasting with hts v4.0
http://feedproxy.google.com/~r/Hyndsight/~3/59E1dgidx54/
Wed, 12 Feb 2014 08:26:42 +0000https://robjhyndman.com/hyndsight/hts4/A new version of my hts package for R is now on CRAN. It was completely re-written from scratch. Not a single line of code survived. There are some minor syntax changes, but the biggest change is speed and scope. This version is many times faster than the previous version and can handle hundreds of thousands of time series without complaining. The speed-up is due to some new research I am doing with Alan Lee (University of Auckland).<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/59E1dgidx54" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/hts4/Detecting seasonality
http://feedproxy.google.com/~r/Hyndsight/~3/Lbx7IM8n1f4/
Sat, 08 Feb 2014 04:08:38 +0000https://robjhyndman.com/hyndsight/detecting-seasonality/I occasionally get email asking how to detect whether seasonality is present in a data set. Sometimes the period of the potential seasonality is known, but in other cases it is not.
I’ve discussed before how to estimate an unknown seasonal period, and how to measure the strength of the seasonality. In this post, I want to look at testing if a series is seasonal when the potential period is known (e.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Lbx7IM8n1f4" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/detecting-seasonality/Interview for the Capital of Statistics
http://feedproxy.google.com/~r/Hyndsight/~3/u7Mrg9Hr5L4/
Wed, 05 Feb 2014 00:00:22 +0000https://robjhyndman.com/hyndsight/cos-interview/Earo Wang recently interviewed me for the Chinese website Capital of Statistics. The English transcript of the intervew is on Earo’s personal website.
This is the third interview I’ve done in the last 18 months. The others were for:
Data Mining Research. Republished in Amstat News.
DecisionStats.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/u7Mrg9Hr5L4" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/cos-interview/Top papers in the International Journal of Forecasting
http://feedproxy.google.com/~r/Hyndsight/~3/o9EX-ww2h2Q/
Tue, 04 Feb 2014 01:06:22 +0000https://robjhyndman.com/hyndsight/ijf-top-papers/Every year or so, Elsevier asks me to nominate five _International Journal of Forecasting_ papers from the last two years to highlight in their marketing materials as “Editor’s Choice”. I try to select papers across a broad range of subjects, and I take into account citations and downloads as well as my own impression of the paper. That tends to bias my selection a little towards older papers as they have had more time to accumulate citations.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/o9EX-ww2h2Q" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/ijf-top-papers/Computational Actuarial Science with R
http://feedproxy.google.com/~r/Hyndsight/~3/95iLCKYdU30/
Mon, 03 Feb 2014 03:00:32 +0000https://robjhyndman.com/hyndsight/caswithr/I recently co-authored a chapter on “Prospective Life Tables” for this book, edited by Arthur Charpentier. R code to reproduce the figures and to complete the exercises for our chapter is now available on github. Code for the other chapters should also be available soon. The book can be pre-ordered on Amazon.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/95iLCKYdU30" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/caswithr/Monash Econometrics in the top 10
http://feedproxy.google.com/~r/Hyndsight/~3/Ndf_hi9w314/
Sun, 02 Feb 2014 01:03:35 +0000https://robjhyndman.com/hyndsight/monash-top10/Dave Giles pointed out on his blog yesterday that my department is currently ranked in the top 10 in the world for econometrics, according to IDEAS. We are also ranked 13th in the world in forecasting. Since IDEAS only covers the economics literature, the forecasting rank does not take account of our work in other areas such as demographic forecasting, and electricity demand forecasting.
These rankings are only a rough indication of quality, but it is nice to see the department being recognized.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Ndf_hi9w314" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/monash-top10/Automatic time series forecasting in Granada
http://feedproxy.google.com/~r/Hyndsight/~3/T_IPNtTkZ64/
Fri, 31 Jan 2014 03:06:24 +0000https://robjhyndman.com/hyndsight/granada-workshop/In two weeks I am presenting a workshop at the University of Granada (Spain) on Automatic Time Series Forecasting.
Unlike most of my talks, this is not intended to be primarily about my own research. Rather it is to provide a state-of-the-art overview of the topic (at a level suitable for Masters students in Computer Science). I thought I’d provide some historical perspective on the development of automatic time series forecasting, plus give some comments on the current best practices.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/T_IPNtTkZ64" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/granada-workshop/Free books on statistical learning
http://feedproxy.google.com/~r/Hyndsight/~3/1eJGP1j8-Y4/
Thu, 30 Jan 2014 01:51:07 +0000https://robjhyndman.com/hyndsight/free-books-on-statistical-learning/Hastie, Tibshirani and Friedman’s Elements of Statistical Learning first appeared in 2001 and is already a classic. It is my go-to book when I need a quick refresher on a machine learning algorithm. I like it because it is written using the language and perspective of statistics, and provides a very useful entry point into the literature of machine learning which has its own terminology for statistical concepts. A free downloadable pdf version is available on the website.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/1eJGP1j8-Y4" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/free-books-on-statistical-learning/Time series data in R
http://feedproxy.google.com/~r/Hyndsight/~3/LPzl0o2Cu-A/
Tue, 28 Jan 2014 22:21:19 +0000https://robjhyndman.com/hyndsight/time-series-data-in-r/There is no shortage of time series data available on the web for use in student projects, or self-learning, or to test out new forecasting algorithms. It is now relatively easy to access these data sets directly in R.M Competition data The 1001 series from the M-competition and the 3003 series from the M3-competition are available as part of the Mcomp package in R.
DataMarket and Quandl Both DataMarket and Quandl contain many thousands of time series that can be downloaded directly into R.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/LPzl0o2Cu-A" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/time-series-data-in-r/Online collaborative writing
http://feedproxy.google.com/~r/Hyndsight/~3/XXswcIN6I5I/
Tue, 28 Jan 2014 02:18:13 +0000https://robjhyndman.com/hyndsight/online-collaborative-writing/Everyone who has written a paper with another author will know it can be tricky making sure you don’t end up with two versions that need to be merged. The good news is that the days of sending updated drafts by email backwards and forwards is finally over (having lasted all of 25 years – I can barely imagine writing papers before email).LaTeX solutions There has been a lot of activity in the development of online LaTeX tools over the last few years.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/XXswcIN6I5I" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/online-collaborative-writing/New in forecast 5.0
http://feedproxy.google.com/~r/Hyndsight/~3/vA6wWUci34E/
Mon, 27 Jan 2014 02:26:52 +0000https://robjhyndman.com/hyndsight/forecast5/Last week, version 5.0 of the forecast package for R was released. There are a few new functions and changes made to the package, which is why I increased the version number to 5.0. Thanks to Earo Wang for helping with this new version.
Handling missing values and outliers Data cleaning is often the first step that data scientists and analysts take to ensure statistical modelling is supported by good data.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/vA6wWUci34E" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/forecast5/Thoughts on the Ljung-Box test
http://feedproxy.google.com/~r/Hyndsight/~3/hBbc3ygHx6o/
Fri, 24 Jan 2014 03:54:34 +0000https://robjhyndman.com/hyndsight/ljung-box-test/<p>It is common to use a Ljung-Box test to check that the residuals from a time series model resemble white noise. However, there is very little practical advice around about how to choose the number of lags for the test.</p><img src="http://feeds.feedburner.com/~r/Hyndsight/~4/hBbc3ygHx6o" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/ljung-box-test/Looking for a new post-doc
http://feedproxy.google.com/~r/Hyndsight/~3/u_cDaoBvKzE/
Wed, 22 Jan 2014 02:50:49 +0000https://robjhyndman.com/hyndsight/postdoc/We are looking for a new post-doctoral research fellow to work on the project “Macroeconomic Forecasting in a Big Data World”. Details are given at the link below
jobs.monash.edu.au/jobDetails.asp?sJobIDs=519824
This is a two year position, funded by the Australian Research Council, and working with me, George Athanasopoulos, Farshid Vahid and Anastasios Panagiotelis. We are looking for someone with a PhD in econometrics, statistics or machine learning, who is well-trained in computationally intensive methods, and who has a background in at least one of time series analysis, macroeconomic modelling, or Bayesian econometrics.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/u_cDaoBvKzE" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/postdoc/Estimating a nonlinear time series model in R
http://feedproxy.google.com/~r/Hyndsight/~3/oh9_5A7_69k/
Tue, 21 Jan 2014 04:24:48 +0000https://robjhyndman.com/hyndsight/nlts/There are quite a few R packages available for nonlinear time series analysis, but sometimes you need to code your own models. Here is a simple example to show how it can be done.
The model is a first order threshold autoregression:
$$ y_t = \begin{cases} \alpha y_{t-1} + e_t & \text{if $y_{t-1} \le r$}\\ \beta y_{t-1} + \gamma e_t & \text{if $y_{t-1}r$} \end{cases} $$ where $e_t$ is a Gaussian white noise series with variance $\sigma^2$.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/oh9_5A7_69k" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/nlts/Judgmental forecasting experiment
http://feedproxy.google.com/~r/Hyndsight/~3/2-fU1Sz5yVA/
Sun, 22 Dec 2013 06:21:24 +0000https://robjhyndman.com/hyndsight/judgmental-forecasting-experiment/The Centre for Forecasting at Lancaster University is conducting some research on judgmental forecasting and model selection. They hope to compare the performance of judgmental model selection with statistical model selection, in order to learn how to best design forecasting support systems. They would like forecasting students, practitioners and researchers to participate, and are offering £50 Amazon Gift Cards as prizes. Here is a brief description from Fotios Petropoulos: We are inviting you to participate in a web-based judgmental forecasting exercise.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/2-fU1Sz5yVA" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/judgmental-forecasting-experiment/How to get your paper rejected quickly
http://feedproxy.google.com/~r/Hyndsight/~3/A3ALTjrmcPI/
Mon, 25 Nov 2013 01:20:04 +0000https://robjhyndman.com/hyndsight/quick-rejection/I sent this rejection letter this morning about a paper submitted to the International Journal of Forecasting.
Dear XXXXX.
I am writing to you regarding manuscript ????? entitled “xxxxxxxxxxxx” which you submitted to the International Journal of Forecasting.
It so happens that I am aware that this paper was previously reviewed for the YYYYYYY journal. It seems that you have not bothered to make any of the changes recommended by the reviewers of your submission to YYYYYYY.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/A3ALTjrmcPI" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/quick-rejection/Three jobs at Monash
http://feedproxy.google.com/~r/Hyndsight/~3/U0fyr5dUp38/
Fri, 18 Oct 2013 04:02:37 +0000https://robjhyndman.com/hyndsight/three-jobs-at-monash/We are currently advertising for three academic positions, suitable for recent PhD graduates.
Lecturer (Applied Statistics or Operations Research) Five-year position with MAXIMA and the School of Mathematical Sciences
Two positions available.
Applications close 31 October.
More information.
Lecturer (Econometrics/Business Statistics) Continuing position with the Department of Econometrics and Business Statistics
Applications close 31 January 2014.
More information.
Please don’t send any questions to me.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/U0fyr5dUp38" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/three-jobs-at-monash/Probabilistic Energy Forecasting
http://feedproxy.google.com/~r/Hyndsight/~3/aEBqgsZjmz4/
Mon, 14 Oct 2013 04:07:55 +0000https://robjhyndman.com/hyndsight/probabilistic-energy-forecasting/The International Journal of Forecasting is calling for papers on probabilistic energy forecasting. Here are the details (taken from Tao Hong’s blog).In today’s competitive and dynamic environment, more and more decision making processes in the energy industry are relying on probabilistic forecasts. The applications of probabilistic energy forecasts spread across planning and operations of the entire energy value chain. We are seeking papers from researchers working on the areas of probabilistic energy forecasting.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/aEBqgsZjmz4" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/probabilistic-energy-forecasting/Robert G Brown (1923-2013)
http://feedproxy.google.com/~r/Hyndsight/~3/Dv8hD_4W7NA/
Wed, 09 Oct 2013 11:52:08 +0000https://robjhyndman.com/hyndsight/rgbrown/Robert Goodell Brown was the father of exponential smoothing. He died last week at the age of 90. While I never met him, I was indebted to him for exponential smoothing and his practical and insightful books.
Today I received this email from King Harrison III advising of his death. Twenty years ago I attended the ISF 93 conference in Pittsburgh, which honored Bob Brown on his 70th birthday, and his contributions to the forecasting world.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Dv8hD_4W7NA" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/rgbrown/OTexts.org is launched
http://feedproxy.google.com/~r/Hyndsight/~3/YGM4vilr42U/
Thu, 26 Sep 2013 23:08:46 +0000https://robjhyndman.com/hyndsight/otexts/The publishing platform I set up for my forecasting book has now been extended to cover more books and greater functionality. Check it out at www.otexts.org.
So far, we have three complete books:
Forecasting: principles and practice, by Rob J Hyndman and George Athanasopoulos
Statistical foundations of machine learning, by Gianluca Bontempi and Souhaib Ben Taieb
Modal logic of strict necessity and possbibility, by Evgeni Latinov
and one book currently being written:<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/YGM4vilr42U" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/otexts/Forecasting with daily data
http://feedproxy.google.com/~r/Hyndsight/~3/hTTqcrkH36w/
Tue, 17 Sep 2013 00:26:53 +0000https://robjhyndman.com/hyndsight/dailydata/I’ve had several emails recently asking how to forecast daily data in R. Unless the time series is very long, the easiest approach is to simply set the frequency attribute to 7.
y <- ts(x, frequency=7) Then any of the usual time series forecasting methods should produce reasonable forecasts. For example
library(forecast) fit <- ets(y) fc <- forecast(fit) plot(fc) When the time series is long enough to take in more than a year, then it may be necessary to allow for annual seasonality as well as weekly seasonality.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/hTTqcrkH36w" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/dailydata/MAXIMA research centre at Monash Uni
http://feedproxy.google.com/~r/Hyndsight/~3/9-VOGexs_BQ/
Wed, 11 Sep 2013 01:17:02 +0000https://robjhyndman.com/hyndsight/maxima/The “Monash Academy for Cross and Interdisciplinary Mathematical Applications” (MAXIMA) is a new research centre that aims to maximise the potential of mathematics to deliver impact to society. It will be led by Kate Smith-Miles. I will also be involved along with several other mathematicians at Monash. Our mission at MAXIMA is to find solutions to 21st century problems by dismantling mathematical barriers.
MAXIMA will be launched on 25 September at a public lecture on “The Role of Embedded Optimization in Smart Systems and Products”.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/9-VOGexs_BQ" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/maxima/Reflections on UseR! 2013
http://feedproxy.google.com/~r/Hyndsight/~3/FsO4Bbxjc4E/
Fri, 12 Jul 2013 22:01:57 +0000https://robjhyndman.com/hyndsight/user2013/This week I’ve been at the R Users conference in Albacete, Spain. These conferences are a little unusual in that they are not really about research, unlike most conferences I attend. They provide a place for people to discuss and exchange ideas on how R can be used.
Here are some thoughts and highlights of the conference, in no particular order. Håvard Rue spoke on Bayesian computing with INLA and the R-INLA package.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/FsO4Bbxjc4E" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/user2013/Asking good questions
http://feedproxy.google.com/~r/Hyndsight/~3/Ooz7WF4NmIA/
Wed, 10 Jul 2013 21:58:55 +0000https://robjhyndman.com/hyndsight/questions/I’m currently attending my third conference in three weeks. So I’ve heard a lot of talks, and I’ve heard a lot of questions asked after the talks. In this guest post, Eran Raviv reflects on what makes a good question after a talk.
A few weeks back I attended the excellent ISF conference. In one of the sessions, the presenter was talking about a state-of-the-art method to prevent model overfitting, an issue with an ever-growing importance in this era of increased computational power and data storage capabilities.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Ooz7WF4NmIA" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/questions/Facts and fallacies of the AIC
http://feedproxy.google.com/~r/Hyndsight/~3/nZtwPZVItR0/
Wed, 03 Jul 2013 21:19:13 +0000https://robjhyndman.com/hyndsight/aic/Akaike’s Information Criterion (AIC) is a very useful model selection tool, but it is not as well understood as it should be. I frequently read papers, or hear talks, which demonstrate misunderstandings or misuse of this important tool. The following points should clarify some aspects of the AIC, and hopefully reduce its misuse. The AIC is a penalized likelihood, and so it requires the likelihood to be maximized before it can be calculated.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/nZtwPZVItR0" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/aic/End of Google Reader
http://feedproxy.google.com/~r/Hyndsight/~3/A5ZKrBr02oQ/
Sat, 29 Jun 2013 08:19:25 +0000https://robjhyndman.com/hyndsight/feedly/If you are reading this in Google Reader, or you are reading in an RSS reader that uses Google Reader as a back-end, then you probably need this reminder. Everyone else can stop reading now.Google Reader will cease to be available on 1 July and you will lose all your feeds and other data if you have not switched before then.
I have switched to using Feedly and it is a very convenient replacement.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/A5ZKrBr02oQ" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/feedly/Future ISFs
http://feedproxy.google.com/~r/Hyndsight/~3/iF-tZ_JjH2g/
Wed, 26 Jun 2013 07:12:56 +0000https://robjhyndman.com/hyndsight/future-isfs/The next few locations for the International Symposium on Forecasting have been announced:
2014: Rotterdam, The Netherlands
2015: Riverside, California, USA
2016: Santander, Spain
2017: Cairns, Australia
The ISF is easily the best forecasting conference, and is held every year in different locations. The 2013 conference finished today in Seoul, South Korea.
I’m especially delighted that it is coming back to Australia in 2017. It has only once before been held in Australia, in 2004.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/iF-tZ_JjH2g" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/future-isfs/Managing research ideas
http://feedproxy.google.com/~r/Hyndsight/~3/h4BlhzseTRo/
Sat, 25 May 2013 13:14:16 +0000https://robjhyndman.com/hyndsight/managing-research-ideas/I received this email today:
Dear Professor Hyndman, I was wondering if you could maybe give me some advice on how to organize your research process. I am able to search the literature on a certain topic and identify where there is a question to work with. My main difficult is to organize my paper annotations in order to help me to guide my research process, i.e, how to manage the information gathered in those papers to compose and structure a document which can represent the research developed so far.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/h4BlhzseTRo" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/managing-research-ideas/IJF quality indicators
http://feedproxy.google.com/~r/Hyndsight/~3/nhYi0LaqOxI/
Fri, 17 May 2013 00:35:16 +0000https://robjhyndman.com/hyndsight/ijf-quality-indicators/I often receive email asking about IJF quality indicators. Here is one I received today.
Dear Professor Hyndman,
I recently had a paper published in IJF entitled, “xxxxxxxxxxxx”. I am very pleased with the publication and consider IJF to be an excellent outlet for my work in time-series econometrics.
I have an unusual request, but I hope you will consider responding. My research is judged by non-economists and IJF is not on their list of “quality” journals.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/nhYi0LaqOxI" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/ijf-quality-indicators/Forecasting annual totals from monthly data
http://feedproxy.google.com/~r/Hyndsight/~3/pFpAz9QTtRE/
Wed, 15 May 2013 06:39:10 +0000https://robjhyndman.com/hyndsight/forecasting-annual-totals/This question was posed on crossvalidated.com:
I have a monthly time series (for 2009-2012 non-stationary, with seasonality). I can use ARIMA (or ETS) to obtain point and interval forecasts for each month of 2013, but I am interested in forecasting the total for the whole year, including prediction intervals. Is there an easy way in R to obtain interval forecasts for the total for 2013?
I’ve come across this problem before in my consulting work, although I don’t think I’ve ever published my solution.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/pFpAz9QTtRE" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/forecasting-annual-totals/Establishing priority
http://feedproxy.google.com/~r/Hyndsight/~3/_h8lfveLaEw/
Mon, 06 May 2013 03:34:34 +0000https://robjhyndman.com/hyndsight/establishing-priority/The nature of research is that other people are probably working on similar ideas to you, and it is possible that someone will beat you to publishing them.When I was working on my PhD, I discovered another PhD thesis by Iris Yeung at UKC with almost exactly the same title as mine, and published a year earlier. In those days, a copy of a thesis had to be printed from microfiche, and then posted by snail mail.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/_h8lfveLaEw" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/establishing-priority/My new forecasting book is finally finished
http://feedproxy.google.com/~r/Hyndsight/~3/yGhNAfyuDog/
Sun, 21 Apr 2013 00:19:14 +0000https://robjhyndman.com/hyndsight/fpp-2/My new online forecasting book (written with George Athanasopoulos) is now completed. I previously described it on this blog nearly a year ago.
In reality, an online book is never complete, and we plan to continually update it. But it is now at the point where it is suitable for course work, and contains exercises and references.
We hope that users (especially other lecturers) will submit materials such as slides and exercises, that can be shared on the website.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/yGhNAfyuDog" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/fpp-2/George E P Box (1919-2013)
http://feedproxy.google.com/~r/Hyndsight/~3/Tw1pLPrNRBo/
Sun, 31 Mar 2013 05:21:41 +0000https://robjhyndman.com/hyndsight/gepbox/Last Thursday (28 March 2013), George Box passed away at the age of 93. He was one of the great statisticians of the last 100 years, and leaves an astonishingly diverse legacy.
When I teach forecasting to my second year commerce students, we cover Box-Cox transformations, Box-Pierce and Ljung-Box tests, and Box-Jenkins modelling, and my students wonder if it is the same Box in all cases. It is. And we don’t even go near his work on response surface modelling, design of experiments, quality control or random number generation.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Tw1pLPrNRBo" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/gepbox/The difference between prediction intervals and confidence intervals
http://feedproxy.google.com/~r/Hyndsight/~3/Jv7KODo88_M/
Tue, 12 Mar 2013 22:56:55 +0000https://robjhyndman.com/hyndsight/intervals/Prediction intervals and confidence intervals are not the same thing. Unfortunately the terms are often confused, and I am often frequently correcting the error in students’ papers and articles I am reviewing or editing.
A prediction interval is an interval associated with a random variable yet to be observed, with a specified probability of the random variable lying within the interval. For example, I might give an 80% interval for the forecast of GDP in 2014.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Jv7KODo88_M" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/intervals/ETS models now in EViews 8
http://feedproxy.google.com/~r/Hyndsight/~3/LufeVsVr9Yk/
Thu, 28 Feb 2013 22:04:20 +0000https://robjhyndman.com/hyndsight/eviews8/The ETS modelling framework developed in my 2002 IJF paper (with Koehler, Snyder and Grose), and in my 2008 Springer book (with Koehler, Ord and Snyder), is now available in EViews 8. I had no idea they were even working on it, so it was quite a surprise to be told that EViews now includes ETS models.Here is the blurb from the release notes:
EViews 8 now offers support for exponential smoothing using the dynamic nonlinear model framework of Hyndman, Koehler, et al.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/LufeVsVr9Yk" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/eviews8/Removing white space around R figures
http://feedproxy.google.com/~r/Hyndsight/~3/hxuL_WPftLs/
Fri, 22 Feb 2013 00:22:46 +0000https://robjhyndman.com/hyndsight/crop-r-figures/When I want to insert figures generated in R into a LaTeX document, it looks better if I first remove the white space around the figure. Unfortunately, R does not make this easy as the graphs are generated to look good on a screen, not in a document.
There are two things that can be done to fix this problem.First, you can reduce the white space generated by R. I use the following function when saving figures in R.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/hxuL_WPftLs" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/crop-r-figures/Forecasting conferences
http://feedproxy.google.com/~r/Hyndsight/~3/23jlM6PQSXM/
Fri, 15 Feb 2013 01:17:41 +0000https://robjhyndman.com/hyndsight/forecasting-conferences/This year there are no less than three forecasting conferences planned for June and July 2013. As well as the annual International Symposium on Forecasting, there is WIPFOR (Workshop on Industry & Practices for FORecasting) to be held in Clamart (near Paris) in June, and a forecasting stream at the EURO2013 conference in Rome in early July. Some details follow, taken from emails sent to me recently.WIPFOR (Clamart, France, 5-7 June 2013) We would like to bring to your attention the Second Workshop on Industry & Practices for FORecasting (WIPFOR) with an emphasis on Modeling and Stochastic Learning for Forecasting in High Dimension.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/23jlM6PQSXM" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/forecasting-conferences/Hyndsight
http://feedproxy.google.com/~r/Hyndsight/~3/0ScPqFVk4w4/
Thu, 14 Feb 2013 00:09:37 +0000https://robjhyndman.com/hyndsight/hyndsight/Originally, I wrote this blog for my own PhD students and I covered issues to do with research. I called it “Research tips” because that is what it was meant to be.
However, over time I’ve started covering other things of interest to me, and the readership has grown way beyond what I ever expected. So I decided it was time to acknowledge the change of focus with a change of name.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/0ScPqFVk4w4" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/hyndsight/Out-of-sample one-step forecasts
http://feedproxy.google.com/~r/Hyndsight/~3/sDecTT143So/
Wed, 13 Feb 2013 23:30:40 +0000https://robjhyndman.com/hyndsight/out-of-sample-one-step-forecasts/It is common to fit a model using training data, and then to evaluate its performance on a test data set. When the data are time series, it is useful to compute one-step forecasts on the test data. For some reason, this is much more commonly done by people trained in machine learning rather than statistics.
If you are using the forecast package in R, it is easily done with ETS and ARIMA models.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/sDecTT143So" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/out-of-sample-one-step-forecasts/Statistical consulting in Australia
http://feedproxy.google.com/~r/Hyndsight/~3/FghL0AO7Fhs/
Mon, 11 Feb 2013 11:09:59 +0000https://robjhyndman.com/hyndsight/consulting-groups/There must be dozens of statistical consulting businesses and organizations in Australia, each specializing in different areas.
I do some consulting work myself, mostly in the forecasting area, but sometimes in other areas of applied statistics including expert witness work in court cases. Email me if you have a project you would like me to take on. However, I often refer potential clients to other statistical consulting groups, as I only have a limited amount of time I can spend on consulting projects.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/FghL0AO7Fhs" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/consulting-groups/Batch forecasting in R
http://feedproxy.google.com/~r/Hyndsight/~3/3bLXOi4ZPVw/
Mon, 07 Jan 2013 05:13:18 +0000https://robjhyndman.com/hyndsight/batch-forecasting/I sometimes get asked about forecasting many time series automatically. Here is a recent email, for example:
I have looked but cannot find any info on generating forecasts on multiple data sets in sequence. I have been using analysis services for sql server to generate fitted time series but it is too much of a black box (or I don’t know enough to tweak/manage the inputs). In short, what package should I research that will allow me to load data, generate a forecast (presumably best fit), export the forecast then repeat for a few thousand items.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/3bLXOi4ZPVw" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/batch-forecasting/Man vs Wild Data
http://feedproxy.google.com/~r/Hyndsight/~3/z80RxaTCovE/
Fri, 21 Dec 2012 00:13:15 +0000https://robjhyndman.com/hyndsight/ysc2013/I’m speaking on this topic at the Young Statisticians Conference, 7-8 February 2013.
If you’re a young statistician and live in Australia, please book in. It promises to be a great couple of days. Early registrations close on 2 January.
Abstract for my talk:
For 25 years I have been an intrepid statistical consultant, tackling the wild frontiers of real data, real problems and real time constraints. I have faced problems ranging from linguistics to river beds, from making paper plates to selling pies at the MCG, from tax office audits to surveys about the colour purple.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/z80RxaTCovE" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/ysc2013/New in forecast 4.0
http://feedproxy.google.com/~r/Hyndsight/~3/31-d8UgbTxs/
Mon, 03 Dec 2012 03:53:53 +0000https://robjhyndman.com/hyndsight/forecast4/A few days ago I released version 4.0 of the forecast package for R. There were quite a few changes and new features, so I thought it deserved a new version number. I keep a list of changes in the Changelog for the package, but I doubt that many people look at it. So for the record, here are the most important changes to the forecast package made since v3.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/31-d8UgbTxs" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/forecast4/SimpleR tips, tricks and tools
http://feedproxy.google.com/~r/Hyndsight/~3/vfXsdgnROMc/
Tue, 20 Nov 2012 21:09:51 +0000https://robjhyndman.com/hyndsight/simpler/I gave this talk last night to the Melbourne Users of R Network. Examples<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/vfXsdgnROMc" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/simpler/Makefiles for R/LaTeX projects
http://feedproxy.google.com/~r/Hyndsight/~3/HZttr5qV5Z4/
Wed, 31 Oct 2012 02:49:48 +0000https://robjhyndman.com/hyndsight/makefiles/Updated: 21 November 2012
Make is a marvellous tool used by programmers to build software, but it can be used for much more than that. I use make whenever I have a large project involving R files and LaTeX files, which means I use it for almost all of the papers I write, and almost of the consulting reports I produce.If you are using a Mac or Linux, you will already have make installed.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/HZttr5qV5Z4" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/makefiles/YSC2013 funding support
http://feedproxy.google.com/~r/Hyndsight/~3/NoLcA592GGQ/
Mon, 29 Oct 2012 01:25:06 +0000https://robjhyndman.com/hyndsight/ysc2013-funding-support/Young Victorian statisticians should be attending the YSC conference in Melbourne in February 2013. It promises to be a great event, and the speaker line-up looks first-class (apart from one dodgy keynote speaker).
If you think you can’t afford it, there is good news! The local SSA Branch is offering financial support to selected young statisticians. Applicants must be student members of the Victorian Branch (to join, see www.statsoc.org.au/join.htm). Successful applicants will be given free conference registration.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/NoLcA592GGQ" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/ysc2013-funding-support/LaTeX loops
http://feedproxy.google.com/~r/Hyndsight/~3/ukYCRuLtvR8/
Tue, 23 Oct 2012 01:48:47 +0000https://robjhyndman.com/hyndsight/latex-loops/Today I was writing a report which included 20 figures, with the names demandplot1.pdf, demandplot2.pdf, …, demandplot20.pdf, and all with similar captions. Clearly a loop was required. After all, LaTeX is a programming language, so we should be able to take advantage of its capabilities.I found the forloop package which handled the problem perfectly. My first attempt looked like this:
\newcommand{\demandplot}[1]{\begin{figure}\centering \includegraphics[width=\textwidth]{./figs/demandplot#1.pdf} \caption{Hourly demand (GW) for zone #1.} \end{figure}} \newcounter{loop} \forloop{loop}{1}{\value{loop}<21}{\demandplot{\arabic{loop}}} The magic happens in the last line providing a loop from 1 to 20, generating the commands \demandplot{1}, \demandplot{2}, …, \demandplot{20}.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/ukYCRuLtvR8" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/latex-loops/The Young Stats Communication Challenge
http://feedproxy.google.com/~r/Hyndsight/~3/74sJYkcATB0/
Sun, 07 Oct 2012 22:07:54 +0000https://robjhyndman.com/hyndsight/ysc2013comp/The Australian Young Statisticians Conference (Feb 2013) is organizing a communication competition. They invite all early-career statisticians (studying, or within 5 years of graduation) to produce a short (3-5 minute) video for the ABS YSC2013 Video Competition, or a static infographic for the ABS YSC2013 Infographic Competition.
Both competitions have a 1st prize of $500, and 2nd prize of $250.
Entries close 16th November, and winners will be notified by mid-December.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/74sJYkcATB0" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/ysc2013comp/Forecasting research grants
http://feedproxy.google.com/~r/Hyndsight/~3/NYwnu6dU9Dk/
Wed, 03 Oct 2012 00:13:16 +0000https://robjhyndman.com/hyndsight/iifsas/The International Institute of Forecasters and SAS have an annual research grant scheme that has been offered for the past ten years. The amounts offered are small (a total of $10K per year, usually split between 2 or 3 projects), but it might be useful for young researchers wanting a bit of funding to help with their forecasting research. The deadline for 2012 has just been extended to 26 October, which is a sure sign that they don’t have enough applications yet.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/NYwnu6dU9Dk" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/iifsas/Why are some things easier to forecast than others?
http://feedproxy.google.com/~r/Hyndsight/~3/-TSO1_hsTi4/
Tue, 18 Sep 2012 02:15:35 +0000https://robjhyndman.com/hyndsight/hardforecasts/Forecasters are often met with skepticism. Almost every time I tell someone that I work in forecasting, they say something about forecasting the stock market, or forecasting the weather, usually suggesting that such forecasts are hopelessly inaccurate. In fact, forecasts of the weather are amazingly accurate given the complexity of the system, while anyone claiming to forecast the stock market deserves skepticism. So what is the difference between these two types of forecasts, and can we say anything about what can be reasonably be forecast and what can’t?<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/-TSO1_hsTi4" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/hardforecasts/Multivariate time series modelling and forecasting workshop
http://feedproxy.google.com/~r/Hyndsight/~3/tNDBCGE0eDA/
Mon, 03 Sep 2012 06:54:45 +0000https://robjhyndman.com/hyndsight/workshop2013/We are planning a workshop on multivariate time series modelling and forecasting, to be held at Monash University on 18-19 February 2013.
Call for papers: all submissions should be sent in pdf format to george.athanasopoulos@monash.edu by 20 November 2012.
Keynote speakers Professor Helmut Lütkepohl (DIW, Berlin)
Professor Massimiliano Marcellino (European University Institute, Florence)
Associate Professor Andrew Patton (Duke University, Durham NC)
Professor George Kapetanios (Queen Mary, University of London)<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/tNDBCGE0eDA" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/workshop2013/COMPSTAT2012
http://feedproxy.google.com/~r/Hyndsight/~3/iJ_BnwUM4Lk/
Tue, 28 Aug 2012 11:38:28 +0000https://robjhyndman.com/hyndsight/compstat2012/This week I’m in Cyprus attending the COMPSTAT2012 conference. There’s been the usual interesting collection of talks, and interactions with other researchers. But I was struck by two side comments in talks this morning that I’d like to mention.
Stephen Pollock: Don’t imagine your model is the truth Actually, Stephen said something like “economists (or was it econometricians?) have a bad habit of imagining their models are true”. He gave the example of people asking whether GDP “has a unit root”?<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/iJ_BnwUM4Lk" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/compstat2012/Flat forecasts
http://feedproxy.google.com/~r/Hyndsight/~3/7ixRwaCoBGw/
Sun, 19 Aug 2012 23:58:23 +0000https://robjhyndman.com/hyndsight/flat-forecasts/About once a week someone will tell me there is a bug in my forecast package for R because it gives forecasts that are the same for all future horizons. To save answering the same question repeatedly, here is my response.
A point forecast is (usually) the mean of the distribution of a future observation in the time series, conditional on the past observations of the time series. It is possible, even likely in some circumstances, that the future observations will have the same mean and then the forecast function is flat.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/7ixRwaCoBGw" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/flat-forecasts/Interviews
http://feedproxy.google.com/~r/Hyndsight/~3/h5SokkPnlDE/
Thu, 09 Aug 2012 12:57:05 +0000https://robjhyndman.com/hyndsight/interviews/I’ve been interviewed twice in the last year:
For DecisionStats, 9 August 2012.
For Data Mining Research, 21 October 2011. Republished in Amstat News, 1 December 2011.
Some readers of this blog might find them interesting. I said a few things in today’s interview that I don’t think I’ve previously said publicly.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/h5SokkPnlDE" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/interviews/Blogs about research
http://feedproxy.google.com/~r/Hyndsight/~3/5Q-GJ2U3dNY/
Thu, 09 Aug 2012 02:04:38 +0000https://robjhyndman.com/hyndsight/blogs-about-research/If you find this blog helpful (or even if you don’t but you’re interested in blogs on research issues and tools), there are a few other blogs about doing research that you might find useful. Here are a few that I read.
Patter – Pat Thomson.
The Thesis Whisperer – Inger Mewburn.
The Research Whisperer – several RMIT researchers.
the (research) supervisor’s friend – Geof Hill.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/5Q-GJ2U3dNY" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/blogs-about-research/Read the literature
http://feedproxy.google.com/~r/Hyndsight/~3/cpsXTSGrGZM/
Fri, 03 Aug 2012 04:35:58 +0000https://robjhyndman.com/hyndsight/read-the-literature/I’ve just finished another reviewer report for a journal, and yet again I’ve had to make comments about reading the literature. It’s not difficult. Before you write a paper, read what other people have done. A simple search on Google scholar will usually do the trick. And before you submit a paper, check again that you haven’t missed anything important.
The paper I reviewed today did not cite a single reference from either of the two most active research groups in the area in the last ten years.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/cpsXTSGrGZM" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/read-the-literature/Put your pre-prints online
http://feedproxy.google.com/~r/Hyndsight/~3/yE0FKI4xN3I/
Thu, 02 Aug 2012 04:13:20 +0000https://robjhyndman.com/hyndsight/preprints/I have argued previously that research papers should be posted online at the same time as they are submitted to a journal. Sometimes people claim that journals don’t allow it, which is nonsense. Almost every journal allows it, and many also allow the published version of a paper to appear on your personal website.
Today I discovered a new tool (thanks to the IMU newsletter) which makes it easy to check a journal’s policy on this.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/yE0FKI4xN3I" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/preprints/Bare bones beamer
http://feedproxy.google.com/~r/Hyndsight/~3/kyeemqBWxU4/
Wed, 01 Aug 2012 06:17:27 +0000https://robjhyndman.com/hyndsight/beamer/Beamer is far and away the most popular software for presentations amongst researchers in mathematics and statistics. Most conference and seminar talks I attend these days use beamer. Unfortunately, they all look much the same. I think people find beamer themes too hard to modify easily, so a small number of templates get shared around. Even the otherwise wonderful LaTeX Templates site has no beamer examples.
The beamer user guide explains how to make changes but it is not for the faint-hearted (although it is a fantastic resource once you have some expertise).<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/kyeemqBWxU4" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/beamer/Forecasting the Olympics
http://feedproxy.google.com/~r/Hyndsight/~3/YC2dTj--dmU/
Tue, 31 Jul 2012 00:00:08 +0000https://robjhyndman.com/hyndsight/olympics/Forecasting sporting events is a growing research area. The International Journal of Forecasting even had a special issue on sports forecasting a couple of years ago.
The London 2012 Olympics has attracted a few forecasters trying to predict medal counts, world records, etc. Here are some of the articles I’ve seen.
Which Olympic records get shattered?, Nate Silver, New York Times.
Statisticians predict the number of Olympic records that will fall at London 2012, Physics arXiv blog.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/YC2dTj--dmU" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/olympics/Time Series Data Library now on DataMarket
http://feedproxy.google.com/~r/Hyndsight/~3/wKTnWTblE_0/
Wed, 20 Jun 2012 00:30:13 +0000https://robjhyndman.com/hyndsight/tsdl/The Time Series Data Library is a collection of about 800 time series that I have maintained since about 1992, and hosted on my personal website. It includes data from a lot of time series textbooks, as well as many other series that I’ve either collected for student projects or helpful people have sent to me.
I’ve now moved the collection onto DataMarket which provides much better facilities for maintaining and using time series data.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/wKTnWTblE_0" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/tsdl/Constants and ARIMA models in R
http://feedproxy.google.com/~r/Hyndsight/~3/M4NK6PXJPnk/
Wed, 06 Jun 2012 00:07:27 +0000https://robjhyndman.com/hyndsight/arimaconstants/This post is from my new book Forecasting: principles and practice, available freely online at OTexts.org/fpp/.
A non-seasonal ARIMA model can be written as \begin{equation}\label{eq:c} (1-\phi_1B - \cdots - \phi_p B^p)(1-B)^d y_t = c + (1 + \theta_1 B + \cdots + \theta_q B^q)e_t \end{equation} or equivalently as \begin{equation}\label{eq:mu} (1-\phi_1B - \cdots - \phi_p B^p)(1-B)^d (y_t - \mu t^d/d!) = (1 + \theta_1 B + \cdots + \theta_q B^q)e_t, \end{equation} where $B$ is the backshift operator, $c = \mu(1-\phi_1 - \cdots - \phi_p )$ and $\mu$ is the mean of $(1-B)^d y_t$.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/M4NK6PXJPnk" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/arimaconstants/My new forecasting textbook
http://feedproxy.google.com/~r/Hyndsight/~3/weVTdc_noOQ/
Wed, 23 May 2012 01:34:03 +0000https://robjhyndman.com/hyndsight/fpp/After years of saying that I was going to write a book to replace Makridakis, Wheelwright and Hyndman (1998), I’m finally ready to make an announcement!
My new book is Forecasting: principles and practice, co-authored with George Athanasopoulos. It is available online and free-of-charge. We have written about 2⁄3 of the book so far (all of which is already available online), and we plan to finish it by the end of 2012.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/weVTdc_noOQ" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/fpp/Blog aggregators
http://feedproxy.google.com/~r/Hyndsight/~3/SgCeNBGbuu8/
Tue, 15 May 2012 10:34:39 +0000https://robjhyndman.com/hyndsight/blog-aggregators/A very useful way of keeping up with blogs in a particular area is to subscribe to a blog aggregator. These will syndicate posts from a large number of blogs and provide links back to the original sources. So you only need to subscribe once to get all the good stuff in that area.
There are now several blog aggregators available that might be of interest to readers here. And this blog is now syndicated on several other sites including those listed below.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/SgCeNBGbuu8" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/blog-aggregators/Seeking help
http://feedproxy.google.com/~r/Hyndsight/~3/O9qTi9XONps/
Tue, 08 May 2012 13:23:29 +0000https://robjhyndman.com/hyndsight/seeking-help/Every day I receive emails, or comments on this blog, asking for help with R, forecasting, LaTeX, possible research topics, how to install software, or some other thing I’m supposed to know something about. Unfortunately, I cannot provide a one-man help service to the rest of the world. I used to reply to all the requests explaining where to go for help, but I stopped replying a while ago as it took too much time to do even that.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/O9qTi9XONps" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/seeking-help/Measuring time series characteristics
http://feedproxy.google.com/~r/Hyndsight/~3/XEdAp1rmfJI/
Wed, 02 May 2012 07:51:05 +0000https://robjhyndman.com/hyndsight/tscharacteristics/A few years ago, I was working on a project where we measured various characteristics of a time series and used the information to determine what forecasting method to apply or how to cluster the time series into meaningful groups. The two main papers to come out of that project were:
Wang, Smith and Hyndman (2006) Characteristic-based clustering for time series data. Data Mining and Knowledge Discovery, 13(3), 335-364.
Wang, Smith-Miles and Hyndman (2009) “Rule induction for forecasting method selection: meta-learning the characteristics of univariate time series”, Neurocomputing, 72, 2581-2594.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/XEdAp1rmfJI" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/tscharacteristics/LaTeX templates
http://feedproxy.google.com/~r/Hyndsight/~3/16TQlKU0h0E/
Mon, 09 Apr 2012 01:42:16 +0000https://robjhyndman.com/hyndsight/latex-templates/Some of the most popular pages on this site are my LaTeX templates: for a curriculum vitae, a beamer poster, a beamer talk, a Monash University working paper, and a Monash University thesis. Almost all new LaTeX users begin with templates, so it is surprising that there aren’t more good templates around to get people started.
Now there is a great new website for LaTeX templates: www.latextemplates.com. There are some nice templates for letters, lab reports, calendars, theses, assignments, essays, and CVs.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/16TQlKU0h0E" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/latex-templates/Google scholar metrics
http://feedproxy.google.com/~r/Hyndsight/~3/uJEAQKHsLe4/
Mon, 02 Apr 2012 02:45:30 +0000https://robjhyndman.com/hyndsight/google-scholar-metrics/Google has produced another great tool for researchers, this time providing some metrics on journal citations. Google Scholar Metrics allows you to search on journals rather than articles, and to see the average or median h-index for each journal.
For example, a search on “forecasting” yields the following results:
The h-index is the largest number $h$ such that at least $h$ articles in that publication were cited at least $h$ times each.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/uJEAQKHsLe4" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/google-scholar-metrics/Data visualization
http://feedproxy.google.com/~r/Hyndsight/~3/t3UxYEZg7nU/
Sun, 04 Mar 2012 22:37:45 +0000https://robjhyndman.com/hyndsight/data-visualization/For those who have not read the seminal works of Tufte and Cleveland, please hang your heads in shame. To salvage some sense of self-worth, you can then head over to Solomon Messing’s blog where he is starting a series on data visualization based on the principles developed by Tufte and Cleveland (with R examples).
The classics are also worth reading, and remain relevant despite the 20 or 30 years that have elapsed since they appeared.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/t3UxYEZg7nU" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/data-visualization/Exponential smoothing and regressors
http://feedproxy.google.com/~r/Hyndsight/~3/ATSEpfsGi1k/
Tue, 28 Feb 2012 10:15:03 +0000https://robjhyndman.com/hyndsight/ets-regressors/I have thought quite a lot about including regressors (i.e. covariates) in exponential smoothing (ETS) models, and I have done it a couple of times in my published work. See my 2008 exponential smoothing book (chapter 9) and my 2008 Tourism Management paper. However, there are some theoretical issues with these approaches, which have come to light through the research of Ahmad Farid Osman, one of our PhD students at Monash University.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/ATSEpfsGi1k" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/ets-regressors/Academia StackExchange
http://feedproxy.google.com/~r/Hyndsight/~3/4AxrJxNQqBQ/
Wed, 22 Feb 2012 01:04:44 +0000https://robjhyndman.com/hyndsight/academia/There’s a new StackExchange site that might be useful to readers: Academia. It is a Q&A site for academics and those enrolled in higher education.
The draft FAQ says it will cover:
Life as a graduate student, postdoctoral researcher, university professor
Transitioning from undergraduate to graduate researcher
Inner workings of research departments
Requirements and expectations of academicians
Judging from the first 89 questions, this is going to be extremely helpful, especially for PhD students.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/4AxrJxNQqBQ" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/academia/Mailing lists
http://feedproxy.google.com/~r/Hyndsight/~3/t2pemSqEX68/
Thu, 16 Feb 2012 00:25:25 +0000https://robjhyndman.com/hyndsight/mailing-lists/Staying in touch with other researchers is important. You need to know about up-coming conferences, seminars, jobs, etc. To this end, it is worth finding a few key email lists to join. A long list of lists in econometrics and statistics is provided by EconometricLinks. Browse through it to find topics of interest. No doubt researchers in other disciplines have their own lists, but I don’t know about them.
For those in Australia and New Zealand, there are two local lists that every statistician and econometrician should be on:<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/t2pemSqEX68" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/mailing-lists/Table design
http://feedproxy.google.com/~r/Hyndsight/~3/tY0luOeabZw/
Mon, 13 Feb 2012 23:15:09 +0000https://robjhyndman.com/hyndsight/tables/Almost every research paper and thesis in statistics contains at least some tables, yet students are rarely taught how to make good tables. While the principles of good graphics are slowly becoming part of a statistical education (although not an econometrics education!), the principles of good tables are often ignored. Perhaps people think they are obvious, although the results I see in papers and theses suggest otherwise.Lately the topic seems to have been getting some much-needed attention and the following resources may be useful.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/tY0luOeabZw" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/tables/Following authors on Google Scholar
http://feedproxy.google.com/~r/Hyndsight/~3/I3Nl4wfELA0/
Tue, 31 Jan 2012 01:22:09 +0000https://robjhyndman.com/hyndsight/googlescholar2/A great new feature has been added to Google Scholar Citations. For those authors who have set up a citations page, it is now possible to get email alerts for any new articles they publish, or for any new citations of their articles. So you can track citations to your own work this way, and stay up-to-date with key authors in your field.
Setting up a Google Citations page is super-easy and was already worth doing.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/I3Nl4wfELA0" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/googlescholar2/Refereeing a journal article
http://feedproxy.google.com/~r/Hyndsight/~3/oORA6XlIOrQ/
Fri, 20 Jan 2012 02:50:56 +0000https://robjhyndman.com/hyndsight/refereeing2/I’ve written briefly on this before. For an excellent and more detailed discussion of what is involved, there is a series of excellent posts on Pat Thomson’s blog:
Refereeing a journal article part 1: reading
Refereeing a journal article part 2: making a recommendation
Refereeing a journal article part 3: writing the feedback
If every reviewer followed her advice, my life as an editor would be much easier, and the quality of research would be improved.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/oORA6XlIOrQ" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/refereeing2/Internet surveys
http://feedproxy.google.com/~r/Hyndsight/~3/1NZ9MCly-9I/
Wed, 18 Jan 2012 23:30:12 +0000https://robjhyndman.com/hyndsight/surveys/I received the following email today:
I am preparing a thesis … I need to conduct the widest possible poll, and it occurred to me that perhaps you could guide me toward an internet-based way in which this can be done easily. I have a ten-question questionnaire prepared, that I wish to have an random sample of the population respond to. I have no budget for this, so I hope you can suggest a way in which a good number of responses can be harvested using blogs or sites you may be aware of.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/1NZ9MCly-9I" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/surveys/Are we getting better at forecasting?
http://feedproxy.google.com/~r/Hyndsight/~3/q71bRSmjjFo/
Fri, 23 Dec 2011 04:39:02 +0000https://robjhyndman.com/hyndsight/bostonglobe/I was interviewed recently for the Boston Globe. The interview was by email and I thought it might be useful to post here.Here are the questions from the journalist.
Are we better at predicting future events than we used to be? Or are there obstacles inherent to the endeavor that prevent us from ever really being able to do it accurately? If we are better, then what allowed it? Computational power?<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/q71bRSmjjFo" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/bostonglobe/Organizing travel
http://feedproxy.google.com/~r/Hyndsight/~3/KNUYMZLhyCc/
Mon, 19 Dec 2011 13:12:27 +0000https://robjhyndman.com/hyndsight/travel/Whether travelling to a seminar or conference, or just having a holiday, using a travel organizer can make the process simpler and easier. A good travel organizer keeps all your travel details (flights, hotels, car rentals, meetings, weather forecasts, etc.) organized and synced to whatever devices you use (two computers, an iPad and an iPhone in my case).I am aware of four travel organizers that do this: TripIt, TripCase, WorldMate and Wipolo.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/KNUYMZLhyCc" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/travel/Forecasting time series using R
http://feedproxy.google.com/~r/Hyndsight/~3/vFabZHso9pk/
Fri, 16 Dec 2011 02:30:30 +0000https://robjhyndman.com/hyndsight/melburntalk/I gave this talk on Forecasting time series using R for the Melbourne Users of R Network (MelbURN) on Thursday 27 October 2011.
Slides Examples
Abstract I look at the various facilities for time series forecasting available in R, concentrating on the forecast package. This package implements several automatic methods for forecasting time series including forecasts from ARIMA models, ARFIMA models and exponential smoothing models. I also look more generally at how to go about forecasting non-seasonal data, seasonal data, seasonal data with high frequency, and seasonal data with multiple frequencies.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/vFabZHso9pk" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/melburntalk/Cyclic and seasonal time series
http://feedproxy.google.com/~r/Hyndsight/~3/fn_3g3wqL0Q/
Wed, 14 Dec 2011 05:22:10 +0000https://robjhyndman.com/hyndsight/cyclicts/These terms get confused all the time (e.g., this question on CrossValidated.com), and so I thought it might be helpful to try to summarize the distinction and some of the associated models.Definitions A seasonal pattern exists when a series is influenced by seasonal factors (e.g., the quarter of the year, the month, or day of the week). Seasonality is always of a fixed and known period. Hence, seasonal time series are sometimes called periodic time series.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/fn_3g3wqL0Q" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/cyclicts/The art of R programming
http://feedproxy.google.com/~r/Hyndsight/~3/zk22JmShiwY/
Wed, 30 Nov 2011 03:52:51 +0000https://robjhyndman.com/hyndsight/matloff/This is a gem of a book. It will become the book I give PhD students when they are learning how to write good R code. That is, if I ever see it again. I had hoped to write a review of it, but I haven’t seen it since it arrived in the mail a couple of weeks ago because a research student or research assistant has always had it on loan.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/zk22JmShiwY" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/matloff/Kaggle on TV
http://feedproxy.google.com/~r/Hyndsight/~3/9B9Cd5uUOnM/
Mon, 28 Nov 2011 23:08:53 +0000https://robjhyndman.com/hyndsight/kaggle-on-tv/It is good to see forecasting algorithms getting some mainstream exposure on ABC Catalyst.
Update: See also this great talk by Jeremy Howard, a data scientist from Melbourne and now part of Kaggle.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/9B9Cd5uUOnM" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/kaggle-on-tv/Researcher portals
http://feedproxy.google.com/~r/Hyndsight/~3/bJPMn4nzFQ4/
Sat, 26 Nov 2011 04:14:12 +0000https://robjhyndman.com/hyndsight/portals/A researcher portal is a website that attempts to list all the publications of a given researcher. Some portals also allow sharing papers, interacting with other researchers, calculating citation statistics, etc. Every researcher wants their work read and cited, so these websites can be useful tools for getting your work noticed. They can also function as a de facto home page if you don’t already have a personal website. Conversely, they can be a good way to find new work by researchers in your field.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/bJPMn4nzFQ4" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/portals/What you wish you knew before you started a PhD
http://feedproxy.google.com/~r/Hyndsight/~3/MYmMKwa0lMU/
Fri, 11 Nov 2011 12:34:21 +0000https://robjhyndman.com/hyndsight/wishlist/I asked my research group recently what they wished they had learned before they started work on a PhD. Here are some of the responses. More mathematics. Particular topics they named included real analysis, functional analysis, measure theory, algebra, linear algebra. That would have been my response also. I still wish I knew more mathematics than I do. I did quite a lot of mathematics as an undergraduate, but every year I need to learn some more.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/MYmMKwa0lMU" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/wishlist/Help for forecasting practitioners
http://feedproxy.google.com/~r/Hyndsight/~3/lp8pg8t-1WU/
Sun, 25 Sep 2011 13:19:27 +0000https://robjhyndman.com/hyndsight/forecasting-help/I often get email from forecasters wanting assistance. As much as I’d like to provide a free forecasting advice service to the world, that’s not what I’m paid to do, and I choose to spend my unpaid time on other things. However, there are some very helpful resources available for forecasting practitioners.First, every practicing forecaster should be reading Foresight. It is far and away the best journal or magazine for forecast practitioners.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/lp8pg8t-1WU" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/forecasting-help/The scourge of the academic publishers
http://feedproxy.google.com/~r/Hyndsight/~3/er2gMYyWcOc/
Tue, 30 Aug 2011 05:35:52 +0000https://robjhyndman.com/hyndsight/scourge/Academic publishing is built on an old model where publishers were needed to print and distribute journals to libraries. Under this system, it makes sense that the journals are distributed by publishing companies who charge fees for their work. On the other hand, the academics who write for the journals, the peer reviewers and (almost all) editors, have always contributed their time and expertise without cost. Essentially, they are being paid by universities and other research organizations to do this work.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/er2gMYyWcOc" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/scourge/Time series cross-validation: an R example
http://feedproxy.google.com/~r/Hyndsight/~3/AYCMuDA9yjo/
Fri, 26 Aug 2011 05:23:19 +0000https://robjhyndman.com/hyndsight/tscvexample/I was recently asked how to implement time series cross-validation in R. Time series people would normally call this “forecast evaluation with a rolling origin” or something similar, but it is the natural and obvious analogue to leave-one-out cross-validation for cross-sectional data, so I prefer to call it “time series cross-validation”.Here is some example code applying time series CV and comparing 1-step, 2-step, …, 12-step forecasts using the Mean Absolute Error (MAE).<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/AYCMuDA9yjo" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/tscvexample/Major changes to the forecast package
http://feedproxy.google.com/~r/Hyndsight/~3/daKmPdpFqfI/
Fri, 26 Aug 2011 00:20:04 +0000https://robjhyndman.com/hyndsight/forecast3/The forecast package for R has undergone a major upgrade, and I’ve given it version number 3 as a result. Some of these changes were suggestions from the forecasting workshop I ran in Switzerland a couple of months ago, and some have been on the drawing board for a long time. Here are the main changes in version 3, plus a few earlier additions that I thought deserved a mention.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/daKmPdpFqfI" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/forecast3/Crowd sourcing forecasts
http://feedproxy.google.com/~r/Hyndsight/~3/LSzN0fQEzLk/
Wed, 24 Aug 2011 04:09:22 +0000https://robjhyndman.com/hyndsight/crowd-sourcing-forecasts/Forecasting Ace is looking for participants to develop improved methods for predicting future events and outcomes. Their goal is to develop methods for aggregating many individual judgments in a manner that yields more accurate predictions than any one person or small group alone could provide. Potential applications of the system include forecasting economic conditions, political changes, technological development and medical breakthroughs. They are currently seeking to recruit individuals who have interest (and/or expertise) in any of a wide range of current affairs, including but not limited to politics, economics, technology, military affairs, social trends, and science and technology.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/LSzN0fQEzLk" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/crowd-sourcing-forecasts/Learn Machine Learning at Stanford for free
http://feedproxy.google.com/~r/Hyndsight/~3/orGXQHem-aA/
Tue, 16 Aug 2011 06:12:29 +0000https://robjhyndman.com/hyndsight/stanford-ml/Andrew Ng’s machine learning course at Stanford is being offered free to anyone online in the (northern) fall of 2011. I’ve seen some of the notes from this course and it looks to be an excellent broad introduction to machine learning and data mining. For example, support vector machines, neural networks, kernels, clustering, dimension reduction, etc.Statisticians should know something about this area (just as computer scientists working in machine learning should know some statistical modelling), and this would be a great way to learn it.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/orGXQHem-aA" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/stanford-ml/Beware of junk journals and publishers
http://feedproxy.google.com/~r/Hyndsight/~3/oDRYJACs6AE/
Fri, 12 Aug 2011 02:14:05 +0000https://robjhyndman.com/hyndsight/junkjournals/Today I received the following email:
Dear Professor,
Antarctica Journal of Mathematics Archimedes Journal of Mathematics Bessel Journal of Mathematics Cayley Journal of Mathematics Diophantus Journal of Mathematics We are charging only $3 per page, which is very cheap when compared to some money oriented journals.
Further we request you to withdraw your paper from other journals keeping in view of high page charges.
You can submit your research papers to our online journals.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/oDRYJACs6AE" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/junkjournals/Recommended survey papers
http://feedproxy.google.com/~r/Hyndsight/~3/UPoUuiePngA/
Mon, 25 Jul 2011 12:01:06 +0000https://robjhyndman.com/hyndsight/surveypapers/Survey articles are particularly helpful in getting a foothold in a new research area, or in looking for important papers that you may have overlooked. Whatever area of research you are in, look out for survey papers and journals dedicated to publishing survey papers.The journal Statistics Surveys is relatively new (volume 1 in 2007) and provides very helpful survey articles of areas of statistical research. It is sponsored by the American Statistical Association, the Bernoulli Society, the Institute of Mathematical Statistics, and by the Statistical Society of Canada.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/UPoUuiePngA" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/surveypapers/Social networking for researchers
http://feedproxy.google.com/~r/Hyndsight/~3/ej20IVEL-yg/
Thu, 21 Jul 2011 02:39:42 +0000https://robjhyndman.com/hyndsight/socialnetworks/It would be nice to have a place to share ideas, links, comments in a very informal way with others involved in research in statistical methodology and data science. CrossValidated.com is great for specific questions, but is not suitable for commenting on papers or sharing ideas and links.Although I have a facebook account, this seems the wrong place to carry out a research discussion because my family and friends are generally not interested, and it is messy to restrict posts to groups of friends.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/ej20IVEL-yg" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/socialnetworks/Comparing HoltWinters() and ets()
http://feedproxy.google.com/~r/Hyndsight/~3/yApnZLgAEoo/
Sun, 29 May 2011 07:02:01 +0000https://robjhyndman.com/hyndsight/estimation2/I received this email today:
I have a question about the ets() function in R, which I am trying to use for Holt-Winters exponential smoothing. My problem is that I am getting very different estimates of the alpha, beta and gamma parameters using ets() compared to HoltWinters(), and I can’t figure out why.
This is a common question, so I thought the answer might be of sufficient interest to post here.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/yApnZLgAEoo" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/estimation2/I'm switching to TeXstudio
http://feedproxy.google.com/~r/Hyndsight/~3/PhdlT5q9gyo/
Tue, 29 Mar 2011 11:20:13 +0000https://robjhyndman.com/hyndsight/texstudio/I’ve happily used WinEdt for all my LaTeX editing for about 15 years and I’ve encouraged my whole research team to use it. But I’m tired of problems with WinEdt that take up my time.I regularly have requests for help from one of my research team because something in WinEdt is not working properly — such as pdf synchronization problems, or it is using an old version of MikTeX that no longer updates, or that it has switched to using another pdf viewer without warning.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/PhdlT5q9gyo" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/texstudio/Looking after your supervisor
http://feedproxy.google.com/~r/Hyndsight/~3/g9YODz1XF8M/
Fri, 25 Mar 2011 11:17:03 +0000https://robjhyndman.com/hyndsight/looking-after-your-supervisor/Some good advice here: The care and maintenance of your adviser.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/g9YODz1XF8M" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/looking-after-your-supervisor/Ten rules for data analysis
http://feedproxy.google.com/~r/Hyndsight/~3/NMhJisch4fM/
Mon, 14 Mar 2011 23:06:53 +0000https://robjhyndman.com/hyndsight/ten-rules-for-data-analysis/Peter Kennedy was an associate editor of the International Journal of Forecasting and a superb applied econometrician. He died unexpectedly in August 2010. He was best known for his excellent book _A Guide to Econometrics_ as well as his “Ten Commandments of Applied Econometrics”. He provided a variation on his ten commandments in advice to his students in the form of the following ten rules: Use common sense (and economic theory)<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/NMhJisch4fM" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/ten-rules-for-data-analysis/Statistical tests for variable selection
http://feedproxy.google.com/~r/Hyndsight/~3/p-xiDWZiUKA/
Mon, 14 Mar 2011 22:28:09 +0000https://robjhyndman.com/hyndsight/tests2/I received an email today with the following comment:
I’m using ARIMA with Intervention detection and was planning to use your package to identify my initial ARIMA model for later iteration, however I found that sometimes the auto.arima function returns a model where AR/MA coefficients are not significant. So my question is: Is there a way to filter the search for ARIMA models that only have significant coefficients. I can remove the non-significant coefficients but I think it would be better to search for those models that only have significant coefficients.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/p-xiDWZiUKA" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/tests2/RStudio: just what I've been looking for
http://feedproxy.google.com/~r/Hyndsight/~3/BaIyjuMpKGU/
Tue, 01 Mar 2011 05:09:27 +0000https://robjhyndman.com/hyndsight/rstudio/For many years I used RWinEdt as my text editor for R code, but when WinEdt 6.0 came out, RWinEdt stopped working. So I’ve been looking for something to replace it. I’ve tried Tinn-R, NppToR, Eclipse with StatET and a couple of other editors, but nothing was quite right.
Then yesterday, out of the blue, RStudio was announced and it looks fantastic! A screenshot is given below.
The window contains a smart editor with code completion and tabbing, console, workspace with viewable objects, plotting panel with history, etc.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/BaIyjuMpKGU" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/rstudio/Lies, damn lies and statistics
http://feedproxy.google.com/~r/Hyndsight/~3/4sZiwZmlZzc/
Fri, 14 Jan 2011 07:48:36 +0000https://robjhyndman.com/hyndsight/lewandowsky/There’s a nice article with this title by Stephan Lewandowsky on the ABC website today, exploring the difference between anecdotes and data, and the dangers of cherry-picking evidence.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/4sZiwZmlZzc" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/lewandowsky/Six places left for the forecasting workshop
http://feedproxy.google.com/~r/Hyndsight/~3/IPZQRKIU2QE/
Tue, 11 Jan 2011 00:55:43 +0000https://robjhyndman.com/hyndsight/swissworkshop2011a/There are six places left for the forecasting workshop I am giving in Switzerland in June. If you were thinking of going, book in fast!<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/IPZQRKIU2QE" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/swissworkshop2011a/Authorship ethics
http://feedproxy.google.com/~r/Hyndsight/~3/MZBdFvFDkMw/
Tue, 11 Jan 2011 00:42:08 +0000https://robjhyndman.com/hyndsight/authorship/With the constant pressure on academics to publish research papers, there is a temptation for research groups to include “coauthors” who have not really made any contribution to the manuscript. This seems more prevalent in some fields (e.g., the health sciences) than others.
Occasionally, I am asked to add an author to a paper that has already been accepted for publication in the International Journal of Forecasting. I am very reluctant to do this as it is hard to imagine how someone could be left off a paper while it goes through several revisions, only to be remembered after the paper is accepted.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/MZBdFvFDkMw" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/authorship/Becoming a referee
http://feedproxy.google.com/~r/Hyndsight/~3/HMiBuiiFDdQ/
Sun, 09 Jan 2011 22:56:54 +0000https://robjhyndman.com/hyndsight/refereeing/I regularly get emails from people wanting to be referees for the International Journal of Forecasting, usually with an accompanying CV. This is not how the process works.
Referees are almost always selected because they have previously written papers on a similar topic to the manuscript under review. If you want to be a referee, then write good papers and get them published in scholarly journals. Very quickly you will be invited to referee papers in the same journals.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/HMiBuiiFDdQ" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/refereeing/Tips for academic talks
http://feedproxy.google.com/~r/Hyndsight/~3/b9pvruCyHjU/
Sun, 09 Jan 2011 22:21:25 +0000https://robjhyndman.com/hyndsight/talks/There is a nice post on Matt Might’s blog entitled “10 tips on how to give an academic talk”. Check it out.
He recommends the following two books by Joey Asher.
See also my article on “Giving an academic talk”.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/b9pvruCyHjU" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/talks/In praise of Dropbox
http://feedproxy.google.com/~r/Hyndsight/~3/CqwJYAPOHyw/
Thu, 23 Dec 2010 01:49:06 +0000https://robjhyndman.com/hyndsight/dropbox/Every couple of years, a new technology has a big impact on how I work. Gmail was one. My iPhone was another. And I rank Dropbox in the same category.
I get three huge benefits in using Dropbox:
All my files are backed up online. The house can burn down and I know I can still get my files. Also, if I’m away from my desktop or laptop, I can still access my files on my iPhone.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/CqwJYAPOHyw" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/dropbox/CrossValidated Journal Club
http://feedproxy.google.com/~r/Hyndsight/~3/uy3TCJOET4k/
Wed, 22 Dec 2010 00:01:36 +0000https://robjhyndman.com/hyndsight/cvjournalclub/Journal Clubs are a great way to learn new research ideas and to keep up with the literature. The idea is that a group of people get together every week or so to discuss a paper of joint interest. This can happen within your own research group or department, or virtually online.
There is now a virtual journal club operating in conjunction with CrossValidated.com. The first paper discussed was on text data mining.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/uy3TCJOET4k" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/cvjournalclub/Hamming on research
http://feedproxy.google.com/~r/Hyndsight/~3/hYRxfeOsAw8/
Fri, 10 Dec 2010 09:14:32 +0000https://robjhyndman.com/hyndsight/hamming/Richard Hamming was an excellent mathematician who worked at the interface of mathematics and computer science. In 1986 he gave a wonderful talk entitled You and Your Research. Derek Smith on the AMS Graduate Student blog reminded me of it today. If you haven’t read it previously, stop work immediately and read it now.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/hYRxfeOsAw8" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/hamming/Forecasting workshop: Switzerland, June 2011
http://feedproxy.google.com/~r/Hyndsight/~3/2SDsMX1xtXM/
Mon, 06 Dec 2010 08:57:06 +0000https://robjhyndman.com/hyndsight/swissworkshop2011/I will be running a workshop on Statistical Forecasting: Principles and Practice in Switzerland, 20-22 June 2011. Check out the venue: Waldhotel Doldenhorn, Kandersteg! So if you fancy a trip to the beautiful Swiss Alps next June, read on…Outline Forecasting is required in many situations: deciding whether to build another power generation plant in the next five years requires forecasts of future demand; scheduling staff in a call centre next week requires forecasts of call volume; stocking an inventory requires forecasts of stock requirements.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/2SDsMX1xtXM" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/swissworkshop2011/Data visualization videos
http://feedproxy.google.com/~r/Hyndsight/~3/G0pAMQt25xg/
Tue, 30 Nov 2010 04:05:09 +0000https://robjhyndman.com/hyndsight/data-visualization-videos/Probably everyone has seen Hans Rosling’s famous TED talk by now. If not, here it is:
I recently came across a couple of other exceptional talks on data visualization:
Hans Rosling again: ”Let my dataset change your mindset”. If only all statistics lecturers were this dynamic!
David McCandless: “The beauty of data visualization”. Not so exciting as Hans, but some great examples.
And here’s an hour-length documentary hosted by Hans Rosling called “The Joy of Stats”.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/G0pAMQt25xg" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/data-visualization-videos/Initializing the Holt-Winters method
http://feedproxy.google.com/~r/Hyndsight/~3/_6UUt4jXWXQ/
Tue, 30 Nov 2010 03:09:46 +0000https://robjhyndman.com/hyndsight/hw-initialization/The Holt-Winters method is a popular and effective approach to forecasting seasonal time series. But different implementations will give different forecasts, depending on how the method is initialized and how the smoothing parameters are selected. In this post I will discuss various initialization methods.
Suppose the time series is denoted by $y_1,\dots,y_n$ and the seasonal period is $m$ (e.g., $m=12$ for monthly data). Let $\hat{y}_{t+h|t}$ be the $h$-step forecast made using data to time $t$.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/_6UUt4jXWXQ" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/hw-initialization/A LaTeX template for a CV
http://feedproxy.google.com/~r/Hyndsight/~3/zVad0bqEyYM/
Sat, 27 Nov 2010 05:00:45 +0000https://robjhyndman.com/hyndsight/cv/Every researcher needs a Curriculum Vitae (Latin for “course of life”) or CV. You will need it for job applications, for annual performance appraisal, and just for keeping track of your publications. A CV typically contains lists of achievements including qualifications, publications, presentations, awards, plus teaching experience.
I’ve created a LaTeX style for a CV to make it easy to produce something that looks good and is easy to maintain. You will need an up-to-date implementation of LaTeX because I’m using the wonderful biblatex package (more on that in another post) which has only just become available as part of MikTeX and TeXLive.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/zVad0bqEyYM" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/cv/Tourism forecasting competition ends
http://feedproxy.google.com/~r/Hyndsight/~3/Tq1hkpCPTfg/
Wed, 24 Nov 2010 12:52:00 +0000https://robjhyndman.com/hyndsight/tfc2/And the winners are … Jeremy Howard and Lee C Baker. (See my earlier post for information about the competition.)
Jeremy describes his approach to seasonal time series in a blog post on Kaggle.com. Lee described his approach to annual time series in an earlier post.
A few lessons that come out of this:
For data from a single industry, using a global trend (i.e., estimated across all series) can be useful.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Tq1hkpCPTfg" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/tfc2/Forecast estimation, evaluation and transformation
http://feedproxy.google.com/~r/Hyndsight/~3/vjSUZcTg0lI/
Wed, 10 Nov 2010 00:22:50 +0000https://robjhyndman.com/hyndsight/forecastmse/I’ve had a few emails lately about forecast evaluation and estimation criteria. Here is one I received today, along with some comments.
I have a rather simple question regarding the use of MSE as opposed to MAD and MAPE. If the parameters of a time series model are estimated by minimizing MSE, why do we evaluate the model using some other metric, e.g., MAD and MAPE. I could see that MAPE is not scale dependent.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/vjSUZcTg0lI" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/forecastmse/CrossValidated launched!
http://feedproxy.google.com/~r/Hyndsight/~3/YmF9tW9BtjY/
Thu, 04 Nov 2010 23:45:25 +0000https://robjhyndman.com/hyndsight/crossvalidated/The CrossValidated Q&A site is now out of beta and the new design and site name is live.
New design The new design looks great, thanks to Jin Yang, our designer-in-residence. Note the normal density icon for accepted answers and the site icon depicting a 5-fold cross-validation (light green for the test set and dark green for the training set). There is a faint background graphic in the header and footer from a program that tracks and plots a person’s mouse movement.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/YmF9tW9BtjY" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/crossvalidated/Advice to PhD applicants
http://feedproxy.google.com/~r/Hyndsight/~3/78PCOYK4UTM/
Wed, 03 Nov 2010 04:42:21 +0000https://robjhyndman.com/hyndsight/phdapplicants/For students who are interested in doing a PhD at Monash under my supervision.
First, read the instructions on how to apply.
Second, poke around my website to see the sorts of topics I work on. There’s no point asking to do a PhD with me if you want to do research on something I don’t know much about. In particular, please note that I’m not really interested in finance or economics.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/78PCOYK4UTM" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/phdapplicants/Different results from different software
http://feedproxy.google.com/~r/Hyndsight/~3/uVcXu2FLJnk/
Tue, 26 Oct 2010 08:37:31 +0000https://robjhyndman.com/hyndsight/estimation/I’ve had a few questions on this topic lately. Here is an email received today:
I use Eviews to estimate time series, but I have been checking out R recently, and your Forecast package.
I cannot understand why 2 similar equations in Eviews and R are giving different estimated output. Your insights will be invaluable for my work.
The equations are:
R: Arima(log(fee), order=c(1,1,0), seasonal=list(order=c(1,0,0), period=4), include.drift=TRUE) Eviews: dlog(fee) c ar(1) sar(1) Even with include.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/uVcXu2FLJnk" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/estimation/Getting started with XeLaTeX
http://feedproxy.google.com/~r/Hyndsight/~3/9zvp1d4Fx58/
Sat, 23 Oct 2010 13:30:01 +0000https://robjhyndman.com/hyndsight/xelatex/By now, most LaTeX users have probably heard of XeLaTeX, if only because it is an option in the latest versions of the standard LaTeX editors such as TeXnicCenter, WinEdt and TeXWorks. But most LaTeXers have probably not yet become XeLaTeXers. Why should you?
XeLaTeX is essentially a replacement for pdfLaTeX. It was primarily developed to enable better font handling, especially non-Roman scripts. If you want to write in Telugu, then XeLaTeX is going to make your life much easier.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/9zvp1d4Fx58" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/xelatex/How to avoid annoying a referee
http://feedproxy.google.com/~r/Hyndsight/~3/f4Wlt6xDfdA/
Fri, 22 Oct 2010 09:23:52 +0000https://robjhyndman.com/hyndsight/how-to-avoid-annoying-a-referee/It’s not a good idea to annoy the referees of your paper. They make recommendations to the editor about your work and it is best to keep them happy. There is an interesting discussion on stats.stackexchange.com on this subject. This inspired my own list below.
Explain what you’ve done clearly, avoiding unnecessary jargon. Don’t claim your paper contributes more than it actually does. (I refereed a paper this week where the author claimed to have invented principal component analysis!<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/f4Wlt6xDfdA" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/how-to-avoid-annoying-a-referee/Happy World Statistics Day!
http://feedproxy.google.com/~r/Hyndsight/~3/Vu4wFApYVV8/
Tue, 19 Oct 2010 23:18:14 +0000https://robjhyndman.com/hyndsight/wsd/The United Nations has declared today “World Statistics Day”. I’ve no idea what that means, or why we need a WSD. Perhaps it is because the date is 20.10.2010 (except in North America where it is 10.20.2010). But then, what happens from 2013 to 2099? And do we just forget the whole idea after 3112?
In any case, if we are going to have a WSD, let’s use it to do something useful.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Vu4wFApYVV8" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/wsd/Do something else
http://feedproxy.google.com/~r/Hyndsight/~3/JWmaoAsYT5A/
Sun, 17 Oct 2010 21:36:33 +0000https://robjhyndman.com/hyndsight/do-something-else/I’ve written about taking a break from research before. Along the same lines, there is some good advice on “The Importance of Making Time for “Real World” Activities in Grad School” over at the excellent AMS Graduate Student Blog.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/JWmaoAsYT5A" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/do-something-else/Animated plots in R and LaTeX
http://feedproxy.google.com/~r/Hyndsight/~3/UydHQd764fU/
Wed, 13 Oct 2010 04:45:21 +0000https://robjhyndman.com/hyndsight/animations/I like to use animated plots in my talks on functional time series, partly because it is the only way to really see what is going on with changes in the shapes of curves over time, and also because audiences love them! Here is how it is done.
For LaTeX, you need to create every frame as a separate graphics file. Here is an example. First the R code:
library(demography) nyears <- length(fr.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/UydHQd764fU" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/animations/Always listen to reviewers
http://feedproxy.google.com/~r/Hyndsight/~3/7DwU-xXI-A8/
Wed, 13 Oct 2010 03:39:26 +0000https://robjhyndman.com/hyndsight/always-listen-to-reviewers/This week I was asked to review a paper that I had seen before. It had been submitted to a journal a few months ago and I had written a detailed report describing some problems with the paper, and noting a large number of typos that needed fixing. That journal had rejected the paper, the authors had submitted it to a second journal, and the paper ended up on my desk again for review.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/7DwU-xXI-A8" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/always-listen-to-reviewers/Joining an editorial board
http://feedproxy.google.com/~r/Hyndsight/~3/hwyjlAOQ-V0/
Thu, 07 Oct 2010 05:51:34 +0000https://robjhyndman.com/hyndsight/editorialboards/Being on the editorial board of a journal is a lot of work. I’m currently Editor-in-Chief of the International Journal of Forecasting and previously I’ve been Theory & Methods Editor of the Australian & New Zealand Journal of Statistics. Although it is time-consuming and often goes un-noticed, there are some important rewards that make it worthwhile in my opinion.
You are forced to read carefully a lot of papers in your area of interest.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/hwyjlAOQ-V0" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/editorialboards/Why R is better than Excel for teaching statistics
http://feedproxy.google.com/~r/Hyndsight/~3/4TQH2vRGUfM/
Tue, 05 Oct 2010 05:00:07 +0000https://robjhyndman.com/hyndsight/rvsexcel/This was the topic of a recent conversation on the Australian and New Zealand R mailing list. Here is an edited list of some of the comments made.
R is free.
R is well-documented.
R runs (really well) on *nix as well as Windows and Mac OS.
R is open-source. Trust in the R software is evident by its support among distinguished statisticians. However, the R user need not rely on trust, as the source code for R is freely available for public scrutiny.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/4TQH2vRGUfM" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/rvsexcel/The ARIMAX model muddle
http://feedproxy.google.com/~r/Hyndsight/~3/_UA-CLStCs4/
Mon, 04 Oct 2010 12:39:20 +0000https://robjhyndman.com/hyndsight/arimax/There is often confusion about how to include covariates in ARIMA models, and the presentation of the subject in various textbooks and in R help files has not helped the confusion. So I thought I’d give my take on the issue. To keep it simple, I will only describe non-seasonal ARIMA models although the ideas are easily extended to include seasonal terms. I will include only one covariate in the models although it is easy to extend the results to multiple covariates.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/_UA-CLStCs4" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/arimax/Why every statistician should know about cross-validation
http://feedproxy.google.com/~r/Hyndsight/~3/0E7zlhp8QPE/
Mon, 04 Oct 2010 10:21:11 +0000https://robjhyndman.com/hyndsight/crossvalidation/Surprisingly, many statisticians see cross-validation as something data miners do, but not a core statistical technique. I thought it might be helpful to summarize the role of cross-validation in statistics, especially as it is proposed that the Q&A site at stats.stackexchange.com should be renamed CrossValidated.com.
Cross-validation is primarily a way of measuring the predictive performance of a statistical model. Every statistician knows that the model fit statistics are not a good guide to how well a model will predict: high $R^2$ does not necessarily mean a good model.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/0E7zlhp8QPE" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/crossvalidation/That syncing feeling
http://feedproxy.google.com/~r/Hyndsight/~3/tJSSB28kQDk/
Thu, 30 Sep 2010 06:18:10 +0000https://robjhyndman.com/hyndsight/syncing/Like many people, I use more than one computer and I like to have all my files, bookmarks and other settings synchronized across my computers. Fortunately, that is getting easier as more tools are made available for keeping computers synchronized. I know some of the readers of this blog have been using Xmarks to keep their bookmarks synced, and are now looking at alternatives tools due to the imminent demise of Xmarks.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/tJSSB28kQDk" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/syncing/Forecasting with long seasonal periods
http://feedproxy.google.com/~r/Hyndsight/~3/PSLU5X7KGVA/
Wed, 29 Sep 2010 05:38:51 +0000https://robjhyndman.com/hyndsight/longseasonality/I am often asked how to fit an ARIMA or ETS model with data having a long seasonal period such as 365 for daily data or 48 for half-hourly data. Generally, seasonal versions of ARIMA and ETS models are designed for shorter periods such as 12 for monthly data or 4 for quarterly data.
The ets() function in the forecast package restricts seasonality to be a maximum period of 24 to allow hourly data but not data with a larger seasonal frequency.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/PSLU5X7KGVA" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/longseasonality/Tourism forecasting competition results: part one
http://feedproxy.google.com/~r/Hyndsight/~3/ZVRbduq40z0/
Mon, 20 Sep 2010 09:23:19 +0000https://robjhyndman.com/hyndsight/tfc1/The first stage of the tourism forecasting competition on kaggle has finished. This stage involved forecasting 518 annual time series. Twenty one teams beat our Theta method benchmark which is a great result, and well beyond our expectations. Congratulations to Lee Baker for winning stage one.
I am yet to learn what methods the top teams were using, but we hope to write up a paper for the IJF describing the results.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/ZVRbduq40z0" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/tfc1/Your name is your brand
http://feedproxy.google.com/~r/Hyndsight/~3/Nfu_ckB2hlU/
Thu, 16 Sep 2010 01:59:21 +0000https://robjhyndman.com/hyndsight/names/As a researcher, you want to become known as an expert in your field. You need people to recognize your name and associate it with your research. Consequently, it is important to be consistent in the name you use on publications.
For example, I could write under “R Hyndman”, “R J Hyndman”, “Rob Hyndman”, “Rob J Hyndman”, etc. I’ve chosen the last of these and I try to use it on all publications.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Nfu_ckB2hlU" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/names/How to fail a PhD
http://feedproxy.google.com/~r/Hyndsight/~3/8Efze67Fs8c/
Thu, 02 Sep 2010 13:16:01 +0000https://robjhyndman.com/hyndsight/phdfail/I read an interesting post today by Matt Might on “10 reasons PhD students fail”, and I thought it might be helpful to reflect on some of the barriers to PhD completion that I’ve seen. Matt’s ideas are not all relevant to Australian PhDs, so I have come up with my own list below. Here are the seven steps to failure.
1. Wait for your supervisor to tell you what to do A good supervisor will not tell you what to do.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/8Efze67Fs8c" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/phdfail/Econometrics and R
http://feedproxy.google.com/~r/Hyndsight/~3/-QMTOo3x3jU/
Tue, 31 Aug 2010 01:41:19 +0000https://robjhyndman.com/hyndsight/econometrics-and-r/Econometricians seem to be rather slow to adopt new methods and new technology (compared to other areas of statistics), but slowly the use of R is spreading. I’m now receiving requests for references showing how to use R in econometrics, and so I thought it might be helpful to post a few suggestions here.
A useful on-line and free resource is “Econometrics in R” by Grant Farnsworth. It covers some common econometric methods including heteroskedasticity in regression, probit and logit models, tobit regression, and quantile regression.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/-QMTOo3x3jU" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/econometrics-and-r/Job advertisements
http://feedproxy.google.com/~r/Hyndsight/~3/v742GUlTQSw/
Thu, 26 Aug 2010 01:41:30 +0000https://robjhyndman.com/hyndsight/job-advertisements/Employers often contact me asking how to find a good statistician, econometrician or forecaster for their organization. Students also ask me how to go about finding a job when they finish their degree. This post is for both groups, hopefully making it easier for them to pair up appropriately.
First, the mainstream media outlets are not usually good places to advertise. It seems that few people read printed newspapers anymore.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/v742GUlTQSw" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/job-advertisements/Benchmarks for forecasting
http://feedproxy.google.com/~r/Hyndsight/~3/qdvWQg0sreo/
Wed, 25 Aug 2010 11:03:02 +0000https://robjhyndman.com/hyndsight/benchmarks/Every week I reject papers submitted to the International Journal of Forecasting because they present new methods without ever attempting to demonstrate that the new methods are better than existing methods. It is a policy of the journal that every new method must be compared to standard benchmarks and existing methods before the paper will even be considered for publication.
For univariate time series methods, it is not difficult. As a minimum, comparisons should be made against a naive method and a standard method such as an ARIMA model.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/qdvWQg0sreo" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/benchmarks/Transforming data with zeros
http://feedproxy.google.com/~r/Hyndsight/~3/mTri5gkpzn8/
Fri, 13 Aug 2010 04:22:21 +0000https://robjhyndman.com/hyndsight/transformations/I’m currently working with a hydrologist and he raised a question that occurs quite frequently with real data — what do you do when the data look like they need a log transformation, but there are zero values?
I asked the question on stats.stackexchange.com and received some useful suggestions. What follows is a summary based on these answers, my own experience, plus a few papers I discovered that deal with the topic.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/mTri5gkpzn8" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/transformations/The tourism forecasting competition
http://feedproxy.google.com/~r/Hyndsight/~3/5rnsRgvzarw/
Mon, 09 Aug 2010 05:08:40 +0000https://robjhyndman.com/hyndsight/tourism-forecasting-competition/Recently I wrote a paper entitled “The tourism forecasting competition” in which we (i.e., George Athanasopoulos, Haiyan Song, Doris Wu and I) compared various forecasting methods on a relatively large set of tourism-related time series. The paper has been accepted for publication in the International Journal of Forecasting. (When I submit a paper to the IJF it is always handled by another editor. In this case, Mike Clements handled the paper and it went through several revisions before it was finally accepted.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/5rnsRgvzarw" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/tourism-forecasting-competition/Twenty rules for good graphics
http://feedproxy.google.com/~r/Hyndsight/~3/JQSdnrPB2IE/
Fri, 06 Aug 2010 06:34:14 +0000https://robjhyndman.com/hyndsight/graphics/One of the things I repeatedly include in referee reports, and in my responses to authors who have submitted papers to the International Journal of Forecasting, are comments designed to include the quality of the graphics. Recently someone asked on stats.stackexchange.com aboutbest practices for producing plots. So I thought it might be helpful to collate some of the answers given there and add a few comments of my own taken from things I’ve written for authors.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/JQSdnrPB2IE" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/graphics/Statistical Analysis StackExchange site now available
http://feedproxy.google.com/~r/Hyndsight/~3/RN7jhnIr69I/
Tue, 27 Jul 2010 02:26:39 +0000https://robjhyndman.com/hyndsight/stats-stackexchange/The Q&A site for statistical analysis, data mining, data visualization, and everything else to do with data analysis has finally been launched. Please head over to
stats.StackExchange.com and start asking and answering questions.
Also, spread the word to everyone else who may be interested — work colleagues, students, etc. The more people who use the site, the better it will be. There are already 170 questions, 513 answers and 387 users.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/RN7jhnIr69I" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/stats-stackexchange/More StackExchange sites
http://feedproxy.google.com/~r/Hyndsight/~3/GfAuf3X1314/
Sat, 17 Jul 2010 05:21:41 +0000https://robjhyndman.com/hyndsight/more-stackexchange-sites/The StackExchange site on Statistical Analysis is about to go into private beta testing. This is your last chance to commit if you want to be part of the private beta testing. Don’t worry if you miss out — it will only be a week before it is then open to the public.
There is also a StackExchange site proposal for TeX, LaTeX and friends. Presumably that means that most of the LaTeX questions on StackOverflow will then move to this new site.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/GfAuf3X1314" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/more-stackexchange-sites/The falling standard of English in research
http://feedproxy.google.com/~r/Hyndsight/~3/shcsW9f3dlw/
Thu, 15 Jul 2010 00:53:30 +0000https://robjhyndman.com/hyndsight/english/It seems that most journals no longer do any serious copy-editing, and the standard of English is falling. Today I was reading an article from the European Journal of Operational Research, which is supposedly a good OR journal (current impact factor over 2). Take this for an example from the first page of this paper:
If the learned patterns are unstable, the learning tools would produce inconsistent concepts. To overcome this difficult situation, we employed artificial neural networks (ANNs, NNs) for helping the learning task.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/shcsW9f3dlw" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/english/Academic citations in the popular press
http://feedproxy.google.com/~r/Hyndsight/~3/d4za4891Ijs/
Sun, 11 Jul 2010 19:31:18 +0000https://robjhyndman.com/hyndsight/academic-citations-in-the-popular-press/It is very unusual for a newspaper article to cite an academic paper, unless it is in Nature, Science or the Lancet. Mostly, what we write is too technical and assumes too much background knowledge for it to be accessible to anyone but specialists. So I was pleasantly surprised to find a reference to the International Journal of Forecastingin a recent Wall Street Journalarticle. It is a citation of a 1996 article, so in terms of scientific research it is a bit like quoting the Magna Carta, but a citation nevertheless.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/d4za4891Ijs" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/academic-citations-in-the-popular-press/Use fake data and real data
http://feedproxy.google.com/~r/Hyndsight/~3/CleRoWbAm4w/
Fri, 11 Jun 2010 03:18:34 +0000https://robjhyndman.com/hyndsight/use-fake-data-and-real-data/When developing new statistical methods, it is very useful to test them on both fake data (i.e., simulations) and real data.
Testing on fake data is useful because then you know the “true” answer and can check the procedure under ideal conditions. If your method doesn’t work when the data are designed for the task, it is unlikely to work in real conditions. Fake data also enables you to test the robustness of your method when the conditions aren’t perfect – for example, try adding some nasty outliers and see if the method still works.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/CleRoWbAm4w" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/use-fake-data-and-real-data/Backing up Gmail
http://feedproxy.google.com/~r/Hyndsight/~3/BeD-_yOJrcU/
Thu, 10 Jun 2010 00:59:03 +0000https://robjhyndman.com/hyndsight/backing-up-gmail/I recommend Gmail to everyone who asks, and many who don’t, as it is far superior to every other email platform around. But being paranoid, I don’t like all that valuable email in someone else’s hands. What if Google goes bust one day? Or the Australian government’s internet filter stops gmail? Or I move to China? So I need a local backup just in case. I also need the backup to be painless and not require much attention.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/BeD-_yOJrcU" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/backing-up-gmail/Should you make your working papers public?
http://feedproxy.google.com/~r/Hyndsight/~3/AFB_NUc4LAs/
Wed, 09 Jun 2010 08:05:53 +0000https://robjhyndman.com/hyndsight/working-papers/There seems to be two points of view on this with different practices in different disciplines.
Some researchers do not make their work public until after it has been accepted for publication in a journal. Until that time, drafts of papers are only circulated to close confidants and usually marked “Do not distribute”.
Working papers are published on web sites and in web repositories (such as arXiv or RePEc) as soon as they are finished, at about the same time they are submitted to a journal.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/AFB_NUc4LAs" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/working-papers/Update on a StackExchange site for statistical analysis
http://feedproxy.google.com/~r/Hyndsight/~3/e5UqTiHk2a4/
Wed, 02 Jun 2010 01:53:24 +0000https://robjhyndman.com/hyndsight/stackexchange2/About six weeks ago, I proposed that there should be a Stack Exchange site for questions on data analysis, statistics, data mining, machine learning, etc. I can finally report that there has been substantial progress on this.
The formal proposal is now at Area 51 where the scope of the new site is being developed and voted on in a democratic way. The site has been in a private beta state for a week or so, but is now open for anyone to join in.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/e5UqTiHk2a4" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/stackexchange2/Google scholar alerts
http://feedproxy.google.com/~r/Hyndsight/~3/RpIAJw5iCkg/
Wed, 26 May 2010 06:14:39 +0000https://robjhyndman.com/hyndsight/google-scholar-alerts/A couple of weeks ago, Google scholar added a facility to provide email alerts on new articles associated with specific search queries. First do the search, then click the envelope at top left of screen. For example, here is a search on “exponential smoothing” since 2000.
Note the envelope at the top marked New! Click it to get the following screen.
Those results show some of the flaws in Google Scholar – the dates are not always correct (the first paper listed above appeared in 2004) and there are unresolved duplicates.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/RpIAJw5iCkg" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/google-scholar-alerts/Online mathematical resources
http://feedproxy.google.com/~r/Hyndsight/~3/O_HRlz7engQ/
Thu, 20 May 2010 12:10:00 +0000https://robjhyndman.com/hyndsight/online-mathematical-resources/DLMF For nearly 50 years, a standard reference in mathematical work has been Abramowitz and Stegun’s (1964) Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables. It has provided a marvellous collection of results and tables that have been indispensable for a generation of mathematicians. I’ve used it to look up computationally efficient methods for calculating Bessel functions or gamma functions, or to find one of those trigonometric identities I learned in high school and no longer remember.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/O_HRlz7engQ" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/online-mathematical-resources/Scheduling meetings
http://feedproxy.google.com/~r/Hyndsight/~3/CRzHxaGVEQk/
Sun, 09 May 2010 13:10:27 +0000https://robjhyndman.com/hyndsight/scheduling-meetings/I don’t go to many meetings as I find they are largely a waste of time. In fact, I have the following poster on my office wall to remind everyone who walks in not to ask me to attend a meeting!
But I’m now a chief investigator of an NHMRC grant and I have to meet with other members of the team from time to time. We’ve started using Doodle to schedule our meetings, and it is so good I thought I should share it.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/CRzHxaGVEQk" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/scheduling-meetings/A StackExchange site for statistical analysis?
http://feedproxy.google.com/~r/Hyndsight/~3/gVbR1nAsuhg/
Tue, 20 Apr 2010 08:37:43 +0000https://robjhyndman.com/hyndsight/stackexchange/Regular readers of this site will know I’m a fan of using Stack Overflow for questions about LaTeX, R and other areas of programming. Now the people who produce Stack Overflow are planning on setting up several new sites for asking questions about other topics, and are seeking proposals. I have proposed that there should be a site for questions on data analysis, statistics, data mining, machine learning, etc.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/gVbR1nAsuhg" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/stackexchange/Making a poster in beamer
http://feedproxy.google.com/~r/Hyndsight/~3/fwVTSZxEBWo/
Thu, 25 Mar 2010 04:57:49 +0000https://robjhyndman.com/hyndsight/beamer-poster/This week, I made my first poster. Although I’ve been an academic for more than 20 years, I’ve never had to make a poster before. Some of my coauthors have made posters about our joint research, and two of them have even won prizes (although I can’t take any credit for them). But this week, our department is displaying posters from all research staff about our recent work.
Here is my poster (click for pdf version):<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/fwVTSZxEBWo" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/beamer-poster/My standard LaTeX preamble
http://feedproxy.google.com/~r/Hyndsight/~3/7JiqmuKHe8Y/
Sun, 21 Mar 2010 09:44:12 +0000https://robjhyndman.com/hyndsight/latex-preamble/When I was a PhD student, I found I needed a lot of LaTeX functionality that did not then exist. So I wrote my own package which has served me well for about 20 years. It is called HyTeX.sty (the name being a shameless take-off of LaTeX from Leslie Lamport as well as a homonym of High-Tech). The advantage of having my own package is that almost every file starts with<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/7JiqmuKHe8Y" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/latex-preamble/Using the command line in Windows
http://feedproxy.google.com/~r/Hyndsight/~3/1NkEOBaofcU/
Wed, 10 Mar 2010 02:13:59 +0000https://robjhyndman.com/hyndsight/using-the-command-line-in-windows/Jeromy Anglim is a local blogger who covers a lot of the same territory as this blog. His latest post on running command line programs in Windows is particularly helpful.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/1NkEOBaofcU" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/using-the-command-line-in-windows/Learning R by video
http://feedproxy.google.com/~r/Hyndsight/~3/MygIM8aCYNU/
Wed, 10 Mar 2010 00:10:28 +0000https://robjhyndman.com/hyndsight/learning-r-by-video/For those people who prefer to be shown how to do something rather than read the instructions, there are some videos on using R available online. Here are the ones I know about. Please add links to other similar resources in the comments.
R tour for beginners
R videos
What is R? from Revolution Analytics
R Statistics playlist on YouTube
Statistics with R playlist on YouTube<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/MygIM8aCYNU" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/learning-r-by-video/Top four LaTeX mistakes
http://feedproxy.google.com/~r/Hyndsight/~3/5YyObxZYZRY/
Mon, 15 Feb 2010 23:17:49 +0000https://robjhyndman.com/hyndsight/top-four-latex-mistakes/There is a nice post today by John Cook on the top four LaTeX mistakes. I see these all the time in draft papers by my students and co-authors.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/5YyObxZYZRY" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/top-four-latex-mistakes/What to do when the PhD is finished?
http://feedproxy.google.com/~r/Hyndsight/~3/w-M1W1mB3WI/
Wed, 10 Feb 2010 06:25:22 +0000https://robjhyndman.com/hyndsight/jobs/So you’re wondering what to do once your PhD is finally completed? First, take a holiday. Completing a PhD is an intense and draining exercise, and you should take some time to refresh.
Then you need to decide what career path you prefer. In statistics, there are three choices: the academic route, the semi-academic route and the business route.
Academic route Most PhD students would like an academic job. In fact, I think all of my PhD students have rather fancied my job!<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/w-M1W1mB3WI" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/jobs/Why referee?
http://feedproxy.google.com/~r/Hyndsight/~3/Xiri_QwwAjY/
Wed, 10 Feb 2010 04:25:16 +0000https://robjhyndman.com/hyndsight/why-referee/There are several reasons why researchers should be willing to provide referee reports.
You learn a lot. If the paper is in your area, then writing a referee report forces you to read it very carefully and engage closely with the research of other people in your field. There’s no better way of understand what is going on in your field.
You get better known by the research leaders in your area.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Xiri_QwwAjY" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/why-referee/Writing a referee report
http://feedproxy.google.com/~r/Hyndsight/~3/p5KP2_KpoTo/
Wed, 10 Feb 2010 04:06:38 +0000https://robjhyndman.com/hyndsight/referee-reports/As an editor, I like to see referee reports comprising three sections:
A general summary of the paper and the contribution it makes. You need to highlight here what is new and interesting about the paper, as well as give a summary in a few sentences.
The major problems that need addressing. This is probably the most important section of your report where you explain the main problems. The editor will read this very carefully when deciding whether to accept, reject or invite a revision, so you need to make sure that any problems are clearly explained here.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/p5KP2_KpoTo" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/referee-reports/Using DOIs
http://feedproxy.google.com/~r/Hyndsight/~3/qWW_Ua01d0A/
Sun, 27 Dec 2009 12:35:58 +0000https://robjhyndman.com/hyndsight/doi/Almost all papers these days have a DOI and it is worth knowing how to use them.
At the top or bottom of the first page of a paper, you will see something like this:
doi:10.1016/j.csda.2006.07.028 This is a unique and permanent identifier for the paper known as a “Digital Object Identifier”. The part before the forward slash (10.1016 in the example above) identifies the naming authority (in this case Elsevier) and the part after the forward slash (j.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/qWW_Ua01d0A" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/doi/Replications and reproducible research
http://feedproxy.google.com/~r/Hyndsight/~3/C5blLXBWQD4/
Wed, 02 Dec 2009 06:07:26 +0000https://robjhyndman.com/hyndsight/replications/Reproducible research One of the best ways to get started with research in a new area is to try to replicate some existing research. In doing so, you will usually gain a much better understanding of the topic, and you will often discover some problems with the research, or develop ideas that will lead to a new research paper.
Unfortunately, a lot of papers are not reproducible because the data are not made available, or the description of the methods are not detailed enough.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/C5blLXBWQD4" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/replications/Learning by video
http://feedproxy.google.com/~r/Hyndsight/~3/Bhq8OEmhwmw/
Tue, 24 Nov 2009 02:27:27 +0000https://robjhyndman.com/hyndsight/video/There are some nice online videos available on various aspects of statistics and mathematics that might be helpful to students trying to learn about new areas.
A search on YouTube will lead to a few fairly basic videos.
Statistics playlists Mathematics playlists A better place to go is YouTube EDU which contains material from universities.
Something similar is offered at iTunesU
But the best stuff is on Academic Earth.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Bhq8OEmhwmw" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/video/Controlling figure and table placement in LaTeX
http://feedproxy.google.com/~r/Hyndsight/~3/AdcV-u-Khag/
Wed, 11 Nov 2009 00:01:21 +0000https://robjhyndman.com/hyndsight/latex-floats/It can be frustrating trying to get your figures and tables to appear where you want them in a LaTeX document. Sometimes, they just seem to float off onto another page of their own accord. Here is a collection of tools and ideas that help you get control of those pesky floats.Use the placement options: h, t, b and p. For example
\begin{figure}[htb] causes LaTeX to try to fit the float “here”, or at the “top” of the current page (or the next page), or at the “bottom” of the current page (or the next page).<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/AdcV-u-Khag" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/latex-floats/More on the evils of statistical tests
http://feedproxy.google.com/~r/Hyndsight/~3/kBlq9iCkzpk/
Mon, 09 Nov 2009 23:43:42 +0000https://robjhyndman.com/hyndsight/value-of-p-values/Check out the two posts by Galit Shmueli over at Bzst on hypothesis tests: one on the value of p-values and another on one-sided tests.
She says “Shockingly enough, people seem to really want to use p-values, even if they don’t understand them.” That mirrors my experience too. Confidence intervals are much more useful because they provide a measure of the size of an effect, rather than testing if it is equal to some prespecified value.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/kBlq9iCkzpk" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/value-of-p-values/"Elements of Statistical Learning" now online
http://feedproxy.google.com/~r/Hyndsight/~3/KcX0vloby5U/
Thu, 15 Oct 2009 02:51:18 +0000https://robjhyndman.com/hyndsight/esl2/In the past couple of days, the authors of several blogs have noted that the wonderful book The Elements of Statistical Learning: Data Mining, Inference, and Prediction by Hastie, Tibshirani and Friedman (2nd ed., 2009) is now available for free download in pdf format.
Of course, it is also nice to have a hard copy. Click the image to purchase from Amazon.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/KcX0vloby5U" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/esl2/Using personal pronouns in research writing
http://feedproxy.google.com/~r/Hyndsight/~3/5PaGHAjlDMk/
Thu, 15 Oct 2009 00:49:34 +0000https://robjhyndman.com/hyndsight/personal-pronouns/Should you use “I” or “we” or neither in your thesis or paper?
Thoughts on this have changed over the years. Traditionally, using personal pronouns like “I” and “we” was frowned on. Instead of saying “In Section 3, I have compared the results from method X with those of method Y”, you were expected to write “In section 3, the results from method X are compared with those from method Y”.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/5PaGHAjlDMk" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/personal-pronouns/Attending research seminars
http://feedproxy.google.com/~r/Hyndsight/~3/nwLGiTKzDSc/
Wed, 14 Oct 2009 06:37:20 +0000https://robjhyndman.com/hyndsight/attending-research-seminars/Most research students don’t seem to attend seminars. When asked, they usually say the seminars are not on their topic, or they don’t understand them, or they find them boring, or some other similar reason. I think this is because students don’t understand the purpose of research seminars, and have not learned how to listen to them.
Admittedly, many research seminars are badly presented, and seminar speakers also frequently misunderstand the purpose of the seminar, which makes the problem worse.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/nwLGiTKzDSc" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/attending-research-seminars/Squeezing space with LaTeX
http://feedproxy.google.com/~r/Hyndsight/~3/6E7LChxnECM/
Wed, 14 Oct 2009 05:52:01 +0000https://robjhyndman.com/hyndsight/squeezing-space-with-latex/I’ve been writing a grant application with a 10-page limit, and as usual it is difficult to squeeze everything in. No, I can’t just change the font as it has to be 12 point with at least 2 cm margins on an A4 page. Fortunately, LaTeX is packed full of powerful features that help in squeezing it all in. Here are some of the tips I’ve used over the years.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/6E7LChxnECM" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/squeezing-space-with-latex/Converting eps to pdf
http://feedproxy.google.com/~r/Hyndsight/~3/1Tesao293XI/
Thu, 01 Oct 2009 11:45:42 +0000https://robjhyndman.com/hyndsight/converting-eps-to-pdf/Simply include the package epstopdf. Then when you use pdflatex, the eps files will be automatically converted to pdf at compile time. (The conversion only happens the first time you process the file, and is skipped if there is already a pdf file with the same name.)
For example:
\documentclass{article} \usepackage{graphicx,epstopdf} \begin{document} \includegraphics[width=\textwidth]{fig1} \end{document} Then even though the only graphics file available is fig1.eps, this will still be processed ok using pdflatex or pdftexify.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/1Tesao293XI" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/converting-eps-to-pdf/The 7 secrets of highly successful PhD students
http://feedproxy.google.com/~r/Hyndsight/~3/agMZq2pB5t0/
Mon, 28 Sep 2009 07:04:14 +0000https://robjhyndman.com/hyndsight/7secrets/It seems everyone has 7 secrets to success, and now someone has hopped on the 7-secrets bandwagon with something for PhD students. Thinkwell is an Australian company offering a seminar and associated work book on “The 7 secrets of highly successful PhD students”. I bought the book out of curiosity, but “book” is a gross exaggeration – only eleven pages of fairly simplistic advice. I hope the seminar has more substance.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/agMZq2pB5t0" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/7secrets/Writing an abstract
http://feedproxy.google.com/~r/Hyndsight/~3/Ch6w-ESqlPw/
Sun, 27 Sep 2009 10:19:43 +0000https://robjhyndman.com/hyndsight/abstracts/The abstract is probably the most important part of a paper. Many readers will not read anything else, so you need to grab their attention and get your main message across as clearly and succinctly as possible. It is not meant to be an introduction to the paper, but a summary of the paper. In a single paragraph, a reader can learn the purpose of the research, your general approach to the problem, your main results, and the most important conclusions.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Ch6w-ESqlPw" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/abstracts/Workflow in R
http://feedproxy.google.com/~r/Hyndsight/~3/e3PHcAIkc2g/
Fri, 18 Sep 2009 03:21:33 +0000https://robjhyndman.com/hyndsight/workflow-in-r/This came up recently on StackOverflow. One of the answers was particularly helpful and I thought it might be worth mentioning here. The idea presented there is to break the code into four files, all stored in your project directory. These four files are to be processed in the following order.
load.R This file includes all code associated with loading the data. Usually, it will be a short file reading in data from files.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/e3PHcAIkc2g" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/workflow-in-r/Writing mathematics
http://feedproxy.google.com/~r/Hyndsight/~3/Qr5coJVzZ0A/
Fri, 18 Sep 2009 02:48:39 +0000https://robjhyndman.com/hyndsight/writing-mathematics/Mathematics has its own particular conventions and rules when it comes to writing. There have been several attempts to write them down. The most famous is Halmos’s excellent essay “How to write mathematics”. Other good sources of advice are the following two books:
(I reviewed these two books in the Australian and New Zealand Journal of Statistics a few years ago.)
But if you just want a quick summary, I recommend Dave Richeson’s blog entry “The nuts and bolts of writing mathematics”.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Qr5coJVzZ0A" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/writing-mathematics/Take a break
http://feedproxy.google.com/~r/Hyndsight/~3/O5oTPy7F4Zg/
Sun, 13 Sep 2009 13:06:47 +0000https://robjhyndman.com/hyndsight/take-a-break/Occasionally, the best research is done in long periods of concentrated effort. Allegedly, Isaac Newton used to sometimes write for eight hours standing up without a break.
At other times, taking a break helps the research process. Think of Archimedes and his Eureka moment. Many of my best ideas come while walking, or taking a shower. In fact, I once suggested to my head of department that we should have showers installed in every office as it would increase the quality of our research.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/O5oTPy7F4Zg" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/take-a-break/Finding an R function
http://feedproxy.google.com/~r/Hyndsight/~3/eip0WKn_Oew/
Sun, 13 Sep 2009 11:21:14 +0000https://robjhyndman.com/hyndsight/finding-an-r-function/Suppose you want a function to fit a neural network. What’s the best way to find it? Here are three steps that help to find the elusive function relatively quickly.
First, use help.search("neural") or the shorthand ??neural. This will search the help files of installed packages for the word “neural”. Actually, fuzzy matching is used so it returns pages that have words similar to “neural” such as “natural”. For a stricter search, use help.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/eip0WKn_Oew" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/finding-an-r-function/Statistics education journals
http://feedproxy.google.com/~r/Hyndsight/~3/khSeTc75DPA/
Wed, 02 Sep 2009 03:38:05 +0000https://robjhyndman.com/hyndsight/statistics-education-journals/In many research universities, there can be a tension that arises when great teachers don’t publish much. I believe there is a place for excellent teachers who do limited research within a strong research university, but their contribution is considerably enhanced if they share their teaching insights. There are at least three reputable research journals for publishing articles on statistics education:
Journal of Statistics Education
Statistics Education Research Journal<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/khSeTc75DPA" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/statistics-education-journals/Mathematical research and the internet
http://feedproxy.google.com/~r/Hyndsight/~3/uE_kIZNs2Sw/
Wed, 02 Sep 2009 02:21:54 +0000https://robjhyndman.com/hyndsight/tao-lecture/On Monday night I attended a lecture by Terry Tao on “Mathematical research and the internet”. Terry is Australia’s most famous mathematician, our only Field’s medalist, and one of the most active mathematical bloggers in the world. He has been described as the “Mozart of mathematics” for his remarkable precocity and prolific output. The slides of his talk are available on his blog site.
It was an interesting talk, with excellent slides, marred only by the poor sound system and his bad habit of mumbling.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/uE_kIZNs2Sw" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/tao-lecture/How good are economic forecasts?
http://feedproxy.google.com/~r/Hyndsight/~3/DxDhMsQAWXI/
Thu, 27 Aug 2009 11:47:18 +0000https://robjhyndman.com/hyndsight/how-good-are-economic-forecasts/I wrote last week that “macroeconomic forecasts are little better than shooting blindfold”. I don’t know if it was connected or not, but on the same day a journalist (Richard Pullin) from Reuters phoned me to ask about assessing some economic forecasts. He wanted to compare the accuracy of several economic forecasts for Japan and he wasn’t sure how to go about it. I helped him to calculate the MASE for the different forecasts and the results have now been published.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/DxDhMsQAWXI" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/how-good-are-economic-forecasts/Research supervision workshop
http://feedproxy.google.com/~r/Hyndsight/~3/Mbzk_ijj5vo/
Wed, 26 Aug 2009 06:00:38 +0000https://robjhyndman.com/hyndsight/supervision-workshop/Today I gave a workshop for supervisors of postgraduate students. Mostly I talked about creating a team environment for postgraduate students rather than the traditional model (at least in statistics and econometrics) of each student working in isolation.
The slides are available here in presentation form or in handout form. Actually, these are an edited version of the slides as I accidentally left out a couple of the photographs in the workshop, and I’ve omitted slides that I didn’t end up covering in the workshop.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Mbzk_ijj5vo" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/supervision-workshop/Seek help when it's needed
http://feedproxy.google.com/~r/Hyndsight/~3/Gnm3cIxgvqQ/
Tue, 25 Aug 2009 02:13:37 +0000https://robjhyndman.com/hyndsight/seek-help/I don’t think I’ve had a research student who did not think about giving up at some point. It was part through my second year when I felt like giving up. I felt I was not going to be able to finish my thesis, and that I would be better off throwing in the towel and doing something else. Fortunately, I couldn’t think of anything better to do, plus I hate giving up on anything, so I persevered and it turned out ok.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Gnm3cIxgvqQ" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/seek-help/Why I don't like statistical tests
http://feedproxy.google.com/~r/Hyndsight/~3/jFWXMhwqMJ8/
Mon, 24 Aug 2009 00:40:06 +0000https://robjhyndman.com/hyndsight/tests/It may come as a shock to discover that a statistician does not like statistical tests. Isn’t that what statistics is all about? Unfortunately, in some disciplines statistical analysis does seem to consist almost entirely of hypothesis testing, and therein lies the problem.
The standard practice is to construct a hypothesis test to determine if some attribute of the data is “significant” or not, with the standard p-value threshold of 5%.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/jFWXMhwqMJ8" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/tests/R help on StackOverflow
http://feedproxy.google.com/~r/Hyndsight/~3/01F9pFiutao/
Sun, 23 Aug 2009 22:51:37 +0000https://robjhyndman.com/hyndsight/r-help-on-stackoverflow/Ever since I began using R about ten years ago, the best place to find R help was on the R-help mailing list. But it is time-consuming searching through the archives trying to find something from a long time ago, and there is no way to sort out the good advice from the bad advice.
But now there is a new tool and it is very neat. Head over to the R tag on StackOverflow.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/01F9pFiutao" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/r-help-on-stackoverflow/Backing up
http://feedproxy.google.com/~r/Hyndsight/~3/Tvus4Yj8OAA/
Fri, 21 Aug 2009 03:33:30 +0000https://robjhyndman.com/hyndsight/backing-up/Ever since I deleted the only copy of my honours thesis, one week before it was due to be handed in, I’ve been obsessive about backups, often to the amusement of my family and colleagues. But every time one of them loses a file or has a hard-disk fail, the smiles fade and they ask for advice.
I’ve used many systems over the years, each one a little better than the last.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Tvus4Yj8OAA" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/backing-up/Forecasting the recession
http://feedproxy.google.com/~r/Hyndsight/~3/mGcQMlB2qhY/
Mon, 17 Aug 2009 22:39:27 +0000https://robjhyndman.com/hyndsight/forecasting-the-recession/Forecasters are under the pump with a recession that many didn’t see coming. As I don’t do any macroeconomic forecasting, I can sit back and smile smugly at some of my colleagues while I work on simpler problems such as forecasting in epidemiology, demography and energy demand.
Some of those colleagues are cited in the Wall Street Journal today. The following quotation is interesting:
The spate of cloudy crystal balls highlighted an uncomfortable reality about telling the future: It is hardest when it is most important.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/mGcQMlB2qhY" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/forecasting-the-recession/Maintaining local LaTeX files
http://feedproxy.google.com/~r/Hyndsight/~3/fMHFtRbBZX0/
Fri, 14 Aug 2009 03:46:30 +0000https://robjhyndman.com/hyndsight/localtexmf/If you use LaTeX, then you probably have a bib file — a data base of all the papers and books that you have cited. It is much more efficient to keep one database in one location, than have multiple copies of it floating around your hard drive. (Or even worse, have different bib files created for different papers.) You might also have a few of your own style files, and again it is best to keep these in a central location and not have duplicates all over the place.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/fMHFtRbBZX0" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/localtexmf/Sight what you cite
http://feedproxy.google.com/~r/Hyndsight/~3/xiU_HJ1q-sM/
Thu, 13 Aug 2009 03:46:51 +0000https://robjhyndman.com/hyndsight/sight-what-you-cite/There seems to be a widespread practice of researchers citing papers they have never even seen, let alone read. For example
Some papers claim to do something new when it has already been done in one of the papers cited.
Some articles are cited that apparently have little to do with the reason given for the citation, or which argue the opposite point of view to what is claimed.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/xiU_HJ1q-sM" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/sight-what-you-cite/Songs of Statistics
http://feedproxy.google.com/~r/Hyndsight/~3/xXK9AvTrQ5A/
Wed, 05 Aug 2009 02:57:40 +0000https://robjhyndman.com/hyndsight/songs-of-statistics/If you love statistics (don’t we all?) and can write Chinese (which rules me out), you might like to contribute to the Chinese National Bureau of Statistics celebrations of the 60th anniversary of the “founding of New China”. They are calling for submissions of prose, poetry or song which will “enhance people’s patriotic feelings, statistics and confidence”. Here is an English translation of the page.
Some further translations are on the WSJ page.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/xXK9AvTrQ5A" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/songs-of-statistics/Writing responses to referee reports
http://feedproxy.google.com/~r/Hyndsight/~3/l_Mv3S0IfZE/
Mon, 03 Aug 2009 03:41:47 +0000https://robjhyndman.com/hyndsight/responses/I’ve been spending time writing response letters lately. I’ve also been reading lots of response letters from authors wanting their stuff published in the International Journal of Forecasting. I thought it might be useful to collate a few thoughts on the subject.
No grovelling. I sometimes get response letters that start off with a paragraph of inane and obsequious fawning. The real response only begins after a paragraph of flattery which makes me wonder why I’ve never won the Nobel prize.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/l_Mv3S0IfZE" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/responses/Managing a bibliographic database
http://feedproxy.google.com/~r/Hyndsight/~3/35AQKXVenHw/
Sun, 02 Aug 2009 05:50:11 +0000https://robjhyndman.com/hyndsight/managing-a-bibliographic-database/All researchers need to maintain a database of papers they have read, cited, or simply noted for later reference. For those of us using LaTeX, the database is in the BibTeX format and is stored as a simple text file (a bib file) that can be edited using a text editor such as WinEdt.
But it is often easier to edit the file using specialist software. My current favourite tool is JabRef.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/35AQKXVenHw" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/managing-a-bibliographic-database/Finding LaTeX symbols
http://feedproxy.google.com/~r/Hyndsight/~3/tErMZl3KKwM/
Mon, 27 Jul 2009 05:18:20 +0000https://robjhyndman.com/hyndsight/finding-latex-symbols/All LaTeX users will sometimes need a symbol for which they don’t know the command. If you use WinEdt, there is a neat drop-down menu of some common symbols that can be helpful, but it is necessarily limited. What do you do when you want the male and female symbols?
For several years I’ve been using the comprehensive symbol list whenever I need a symbol in LaTeX that I can’t recall.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/tErMZl3KKwM" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/finding-latex-symbols/Why Word is a bad choice for academic writing
http://feedproxy.google.com/~r/Hyndsight/~3/--wJfOVwPZE/
Tue, 21 Jul 2009 05:20:56 +0000https://robjhyndman.com/hyndsight/why-word-is-a-bad-choice-for-academic-writing/For years I’ve been telling everyone who would listen that MS-Word may sometimes be useful for short notes or for making a “Back in 5 minutes” sign to stick on your door, but if you want to write a serious document like an academic paper, a book or a thesis, then you should use a serious tool such as LaTeX. For those who are not yet convinced, Ben Klemens has a nice article entitled “Why Word is a terrible program”.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/--wJfOVwPZE" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/why-word-is-a-bad-choice-for-academic-writing/Mathematical genealogy
http://feedproxy.google.com/~r/Hyndsight/~3/NkHLbHsfwPY/
Wed, 15 Jul 2009 00:42:07 +0000https://robjhyndman.com/hyndsight/mathematical-genealogy/Having a PhD student is like having a child. I have had many such “children” graduate, and have another few “on the way”. (See here for my offspring.)
Going in the other direction, here is my family tree (or one branch of it), compiled from the Mathematical Genealogy Project. Each successive person is the doctoral student of the person before him. (No females here!)
Erhard Weigel Ph.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/NkHLbHsfwPY" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/mathematical-genealogy/Searching the research literature
http://feedproxy.google.com/~r/Hyndsight/~3/kJSD7_b8KPQ/
Mon, 13 Jul 2009 22:55:56 +0000https://robjhyndman.com/hyndsight/searching-the-research-literature/Most students seem to go to Google first. This is not a good strategy. Google Scholar is much better as it filters out all the junk. Scopus is another engine that aims to do a similar thing. It is better organized but not so complete. ISI WOK is also not as complete as Google Scholar but is particularly good at tracking citations.
Google scholar
Scopus
ISI Web of Knowledge<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/kJSD7_b8KPQ" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/searching-the-research-literature/Clive Granger (1934-2009)
http://feedproxy.google.com/~r/Hyndsight/~3/Zo-Jf8rxcF0/
Mon, 01 Jun 2009 08:27:37 +0000https://robjhyndman.com/hyndsight/granger/Sir Clive Granger has died at the age of 74. There are some nice obituaries in the New York Times and the Daily Telegraph. Also, his Wikipedia page has some good information. I met Clive on several occasions and he was “a scholar and a gentleman”, a remarkably humble man given his outstanding achievements and someone who was always willing to help young researchers. The world of forecasting will miss him.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Zo-Jf8rxcF0" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/granger/Akram's story
http://feedproxy.google.com/~r/Hyndsight/~3/5QSvTdAKxBs/
Thu, 28 May 2009 08:25:58 +0000https://robjhyndman.com/hyndsight/akrams-story/Muhammad Akram was my PhD student a few years ago, and has remained a good friend since he moved on. Here is an interview he recently gave about moving to Australia. Thanks, Akram, for the kind words about me!<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/5QSvTdAKxBs" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/akrams-story/Prediction markets
http://feedproxy.google.com/~r/Hyndsight/~3/UpcaqlJF_PM/
Mon, 18 May 2009 08:24:16 +0000https://robjhyndman.com/hyndsight/prediction-markets/Andrew Leigh has a nice piece in today’s AFR on forecasting via prediction markets<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/UpcaqlJF_PM" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/prediction-markets/Neil Postman on technological change
http://feedproxy.google.com/~r/Hyndsight/~3/VMcZV4gr29M/
Wed, 08 Apr 2009 08:23:00 +0000https://robjhyndman.com/hyndsight/neil-postman-on-technological-change/Neil Postman was Professor of Communication at New York University until his death in 2003. He wrote many wonderfully insightful and thought-provoking articles and books about television, education, technology and childhood. I recently came across a speech he gave in 1998 on “Five things we need to know about technological change”. Here is an online transcript. The five things are:
That we always pay a price for technology; the greater the technology, the greater the price.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/VMcZV4gr29M" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/neil-postman-on-technological-change/Accessing journal articles online
http://feedproxy.google.com/~r/Hyndsight/~3/AhrIGJ2fxqc/
Thu, 12 Mar 2009 08:21:47 +0000https://robjhyndman.com/hyndsight/accessing-journal-articles-online/When searching for research articles online, I often find that the article is unavailable unless I go through the Monash library website, especially when working from home. Here are two solutions to the problem
Within Google scholar, go to “Scholar preferences” and under library links search for “Monash”. Tick the entry “Monash University - Check for full text”. Then save your preferences (button at bottom of page). Next time you do a Google scholar search, a link labelled “Check for full text” will appear beside each entry.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/AhrIGJ2fxqc" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/accessing-journal-articles-online/Statistician: the dream job
http://feedproxy.google.com/~r/Hyndsight/~3/0szvuNVaJXU/
Sun, 01 Feb 2009 08:20:54 +0000https://robjhyndman.com/hyndsight/statistician-the-dream-job/So what is the ultimate job? According to Hal Varian, chief economist at Google, it is being a statistician (see this interview)
Here he is on YouTube with a longer comment:
Statistics - Dream Job of the next decade
From a keynote presentation to the 2008 Almaden Institute - “Innovating with Information”.
The full presentation is available at http://www.almaden.ibm.com/institute/agenda.shtml
Hal Varian makes the argument that with data in huge supply and statisticians in short supply, being a statistician has to be the ‘really sexy job for the 2010s’.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/0szvuNVaJXU" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/statistician-the-dream-job/Time series packages on R
http://feedproxy.google.com/~r/Hyndsight/~3/WA0ETFA_aLo/
Thu, 16 Oct 2008 08:19:39 +0000https://robjhyndman.com/hyndsight/time-series-packages-on-r/There is now an official CRAN Task View for Time Series. This will replace my earlier list of time series packages for R, and provide a more visible and useful entry point for people wanting to use R for time series analysis. If I have missed anything on the list, please let me know.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/WA0ETFA_aLo" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/time-series-packages-on-r/LaTeX tips
http://feedproxy.google.com/~r/Hyndsight/~3/S1YQ0UDl0vg/
Wed, 03 Sep 2008 08:18:19 +0000https://robjhyndman.com/hyndsight/latex-tips/While reading students’ theses and papers recently, I came across various examples of poor latex-ing that I thought would be useful to catalogue.
Don’t set both width and height when using \includegraphics. It distorts the figure. Instead, I suggest using \includegraphics[width=\textwidth]{..}
Be consistent in capitalizing section and subsection headings, and titles in the bibliography. My preferences is to use sentence case throughout.
When using functions such as max, min, log, exp, sin, cos, etc.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/S1YQ0UDl0vg" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/latex-tips/Tracking changes in LaTeX files
http://feedproxy.google.com/~r/Hyndsight/~3/DWjMZ8xAcP8/
Wed, 20 Aug 2008 08:13:47 +0000https://robjhyndman.com/hyndsight/tracking-changes-in-latex-files/When I write a paper, it usually goes through many versions before being submitted to a journal. I keep track of the different versions by renaming the file when I’m about to make major changes, or when I receive a new version from a coauthor. The files are named file1.tex, file2.tex, etc. where “file” is replaced by something more meaningful.
It is often useful to be able to compare versions to see what changes have been made, especially when working with coauthors.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/DWjMZ8xAcP8" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/tracking-changes-in-latex-files/Tracking changes in text files
http://feedproxy.google.com/~r/Hyndsight/~3/PAI5zga27_M/
Wed, 20 Aug 2008 08:09:38 +0000https://robjhyndman.com/hyndsight/tracking-changes-in-text-files/A common issue that arises with text files (e.g., R code) is to identify changes that have been made between versions. I usually number my R files as file1.R, file2.R, etc. (with “file” replaced by something more meaningful),with the number indicating the version of the file. Version numbers change whenever I send the file to someone else to modify, or whenever I make major changes myself.
I often need to know what changes have been made between successive versions.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/PAI5zga27_M" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/tracking-changes-in-text-files/Supervision award
http://feedproxy.google.com/~r/Hyndsight/~3/KrA2zT9t_iw/
Tue, 19 Aug 2008 09:51:00 +0000https://robjhyndman.com/hyndsight/supervision-award/Last night I received the Vice-Chancellor’s postgraduate supervision award at a function at Government House. I am deeply honoured that my students thought to nominate me for the award. I think I was as surprised as anyone to win, and some people have asked me what I did to deserve it. Actually, I’m not sure that I did deserve it, but I can tell you what I told the award committee who chose me.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/KrA2zT9t_iw" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/supervision-award/LaTeX books
http://feedproxy.google.com/~r/Hyndsight/~3/t343eEynWio/
Wed, 30 Jul 2008 10:52:47 +0000https://robjhyndman.com/hyndsight/latex-books/Amazon.com Widgets If you cannot see any books above, please turn off your ad-blocker.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/t343eEynWio" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/latex-books/Time management
http://feedproxy.google.com/~r/Hyndsight/~3/6YryD0b2XwU/
Fri, 25 Jul 2008 08:04:55 +0000https://robjhyndman.com/hyndsight/time-management/I am frequently asked how I manage my time, and how I manage to get so much done. I don’t know that my approach is right for everyone, but in case it helps here are some comments on how I work.
One of the main traps that people fall into is to do things that are urgent, but not necessarily important. They react to the urgency of deadlines, and demands of colleagues, and end up spending a lot of time on things that don’t really matter much.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/6YryD0b2XwU" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/time-management/LaTeX workshop
http://feedproxy.google.com/~r/Hyndsight/~3/gkKHVU89l1w/
Fri, 06 Jun 2008 00:00:00 +0000https://robjhyndman.com/hyndsight/latex-workshop/I gave a one-day LaTeX workshop today.
Here is the blurb:
LaTeX is an extremely powerful markup language for creating structured documents. It is particularly well-suited for documents containing mathematics, but can be used for any document. Those whose publications involve a large number of mathematical equations often use LaTeX rather than MS-Word or some other word processing package. It is the standard writing tool for most research in the mathematical sciences.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/gkKHVU89l1w" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/latex-workshop/Words to avoid
http://feedproxy.google.com/~r/Hyndsight/~3/PpiV9aYRxOI/
Wed, 28 May 2008 08:03:45 +0000https://robjhyndman.com/hyndsight/words-to-avoid/According to Andrew Gelman, we should avoid these words in research writing:
Note that
Interestingly
Obviously
It is clear that
It is interesting to note that
very
quite
of course
Notice that
I agree with him that all of these are overused, but that doesn’t mean they should be banned. The words “very” and “quite” are useful for conveying the strength of a statement.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/PpiV9aYRxOI" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/words-to-avoid/Giving a research seminar
http://feedproxy.google.com/~r/Hyndsight/~3/dj1Gsra7O3k/
Wed, 07 May 2008 07:59:29 +0000https://robjhyndman.com/hyndsight/giving-a-research-seminar/An expanded version of this post is available in my article on “Giving an academic talk”.
With conference season almost upon us, it is timely to discuss what makes a good conference presentation. Here is a suggested structure.
A motivating example demonstrating the problem you are trying to solve. Explain existing approaches to the problem and their weaknesses. Describe your main contributions. Show how your ideas solve the problem/example you started with.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/dj1Gsra7O3k" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/giving-a-research-seminar/Forecasting and time series books
http://feedproxy.google.com/~r/Hyndsight/~3/Ec7Imsm4Mnc/
Mon, 28 Apr 2008 07:56:46 +0000https://robjhyndman.com/hyndsight/forecasting-and-time-series-books/People often ask me for recommendations on forecasting books and time series books. So here is list of ten good books to which I often refer. (If you can’t see a table showing the books below, try turning off your ad-blocker.)
(Updated 23 July 2014)
var amzn_wdgt={widget:'MyFavorites'}; amzn_wdgt.tag='otexts-20'; amzn_wdgt.columns='2'; amzn_wdgt.rows='5'; amzn_wdgt.title='Recommended forecasting and time series books'; amzn_wdgt.width='620'; amzn_wdgt.ASIN='0987507109,3540719164,0324311273,B004K3FKYO,047136164X,0470272848,0792374010,0387953515,144197864X,0387759581'; amzn_wdgt.showImage='True'; amzn_wdgt.showPrice='True'; amzn_wdgt.showRating='False'; amzn_wdgt.design='1'; amzn_wdgt.colorTheme='Pumpkin'; amzn_wdgt.headerTextColor='#eeeeee'; amzn_wdgt.outerBackgroundColor='#31570c'; amzn_wdgt.marketPlace='US'; amzn_wdgt.shuffleProducts='False'; Two are my own books of course (after all, I wrote them because I thought I had something to say).<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Ec7Imsm4Mnc" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/forecasting-and-time-series-books/R workshop
http://feedproxy.google.com/~r/Hyndsight/~3/NJlR1v2Zlcc/
Fri, 28 Mar 2008 07:50:40 +0000https://robjhyndman.com/hyndsight/r-workshop/There was an R workshop on 28-29 June, just before the Australian Statistical Conference. I put in an appearance on the second day giving two talks.
Time series and forecasting in R
handout slides Building R packages for Windows
handout slides<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/NJlR1v2Zlcc" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/r-workshop/Creating a BibTeX file from a Google Library
http://feedproxy.google.com/~r/Hyndsight/~3/1cNeHCzO7YQ/
Thu, 27 Mar 2008 07:54:21 +0000https://robjhyndman.com/hyndsight/creating-a-bibtex-file-from-a-google-library/As you will have seen if you poke around these pages, I have a Google Library of books in statistics and forecasting. This is intended to be a complete copy of what is on the shelves in my office (about 400 books), plus books that I would like on my shelves if I had more space.
I find the library useful for keeping track of books that my students and colleagues borrow (I just add a tag containing their name to the book).<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/1cNeHCzO7YQ" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/creating-a-bibtex-file-from-a-google-library/The maths/stats crisis in Australian education
http://feedproxy.google.com/~r/Hyndsight/~3/kinkv9hXERw/
Wed, 26 Mar 2008 04:50:01 +0000https://robjhyndman.com/hyndsight/the-mathsstats-crisis-in-australian-education/There is a great article in today’s Australian by my co-author Peter Hall on the crisis in mathematics & statistics education (with student numbers falling at the same time as the number of jobs is rising).
The most interesting comment is the last paragraph:
For a nation in the grip of a serious skills shortage in mathematics and statistics, another review is not needed. Action is needed. It should be possible to drive university behaviour in the national interest by foreshadowing that the Government’s proposed compacts will include a requirement that mathematics and statistics curriculums be offered in science and technology, education, business and economics courses across the country, taught by professionals in mathematics and statistics.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/kinkv9hXERw" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/the-mathsstats-crisis-in-australian-education/Dodgy forecasting
http://feedproxy.google.com/~r/Hyndsight/~3/YGWJEYMNQo4/
Thu, 13 Mar 2008 04:48:59 +0000https://robjhyndman.com/hyndsight/dodgy-forecasting/A few years ago I did some forecasting work for a commonwealth government department and found that they were forecasting a $5 billion budget using the FORECAST command in Excel. Worse, they were fitting a regression through only three observations and they were not even the most recent observations.
It seems a similar thing has happened again. The Victorian government is projecting water consumption based on a regression through three observations.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/YGWJEYMNQo4" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/dodgy-forecasting/About Hyndsight
http://feedproxy.google.com/~r/Hyndsight/~3/PbyVGVSKY0s/
Sun, 24 Feb 2008 16:17:53 +0000https://robjhyndman.com/hyndsight/about/I was thinking of writing a book on doing research in statistics. Instead, I decided to write a blog covering the same material, plus other things that might be of interest to my research team. Topics covered include LaTeX, R, writing and preparing a thesis, writing a journal article, submitting an article to a refereed journal, how to convince editors to publish your work, and writing referee reports. Topics of more specific interest to my research team include forecasting, data visualization and functional data, and local events such as meetups or statistics conferences.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/PbyVGVSKY0s" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/about/Indexing in LaTeX
http://feedproxy.google.com/~r/Hyndsight/~3/PhtDZyi9Lmo/
Fri, 16 Nov 2007 04:42:56 +0000https://robjhyndman.com/hyndsight/indexing-in-latex/I’m in the final stages of preparing my new exponential smoothing book for publication and have been learning about some LaTeX indexing tools.
The standard subject index is created using the following procedure:
Include \index{entry} commands wherever you want an index entry.
Include \usepackage{makeidx} and \makeindex in the preamble.
Put a \printindex command where the index is to appear, normally before the \end{document} command.
The details are well-documented in this tutorial (starting on p9).<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/PhtDZyi9Lmo" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/indexing-in-latex/Tables in LaTeX
http://feedproxy.google.com/~r/Hyndsight/~3/syRU5fFzym4/
Thu, 01 Nov 2007 04:25:34 +0000https://robjhyndman.com/hyndsight/tables-in-latex/(Updated May 2017)
Making tables in LaTeX is one of the few areas where LaTeX can be more difficult than a WYSIWYG editor.
Here are some pointers to tools and packages that I have found useful.
tablesgenerator.com: a web-based tool for generating LaTeX tables.
Excel2LaTeX: this excel add-in makes it easy to copy a rectangular array of cells in a spreadsheet into a LaTeX document.
Calc2LaTeX: a similar extension for LibreOffice and OpenOffice.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/syRU5fFzym4" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/tables-in-latex/Searching the statistical literature
http://feedproxy.google.com/~r/Hyndsight/~3/500ci9qRhAo/
Wed, 12 Sep 2007 04:08:55 +0000https://robjhyndman.com/hyndsight/searching-the-statistical-literature/Last week, Google books introduced a facility whereby users can create their own “library” containing a subset of books to search. I have set up a library of statistical books to aid in searching the statistical literature.
My Google library is intended to include all the books on my office shelves, plus a whole lot more that I would like to buy if I had more money (and more shelf space).<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/500ci9qRhAo" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/searching-the-statistical-literature/Organization and R
http://feedproxy.google.com/~r/Hyndsight/~3/rybEQKUt2TY/
Thu, 06 Sep 2007 04:04:54 +0000https://robjhyndman.com/hyndsight/organization-and-r/Many R users seem to get themselves in a bit of a mess with R files and workspaces scattered across different directories. The R files themselves also get messy and hard to follow. So here is some advice on keeping organized with R:
Try to keep code strictly indented based on the code structure such as loops, if statements, etc. Every left brace { should be followed by an extra level of indentation which continues until the matching right brace }.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/rybEQKUt2TY" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/organization-and-r/Forecasting in the news
http://feedproxy.google.com/~r/Hyndsight/~3/YavFN2BqyFM/
Thu, 06 Sep 2007 04:01:29 +0000https://robjhyndman.com/hyndsight/forecasting-in-the-news/There’s an interesting article in today’s Age on the difficulty of long term forecasting. This is an area into which I won’t venture as it requires a strong belief in the idea that the future is just like the observed past.
I am willing to make that assumption in some contexts, and for short-term forecasts, but not in all contexts, and never for long-term forecasts. The article cites the work of Nassim Nicholas Taleb, author of the interesting book The Black Swan, which discusses events which appear highly improbable given the observed history (such as the discovery of black swans in Australia by people who had only ever seen white swans).<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/YavFN2BqyFM" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/forecasting-in-the-news/Debugging in R
http://feedproxy.google.com/~r/Hyndsight/~3/KAGw9R-ctSc/
Fri, 31 Aug 2007 03:53:43 +0000https://robjhyndman.com/hyndsight/debugging-in-r/Anyone who starts writing serious R code (i.e., code that involves user-written functions) soon finds the need to use debugging tools. There are a few basic tools in R that are worth knowing about.
The function debug() allows one to step through the execution of a function, line by line. At any point, you can print out values of variables or produce a graph of the results within the function. While debugging, you can simply type “c” to continue to the end of the current section of code (e.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/KAGw9R-ctSc" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/debugging-in-r/Useful LaTeX links
http://feedproxy.google.com/~r/Hyndsight/~3/Q0e-2_b6diE/
Sun, 30 Jul 2006 10:07:53 +0000https://robjhyndman.com/hyndsight/useful-latex-links/Learning LaTeX Introduction to LaTeX: notes and other materials from a one-day workshop.
Getting started with LaTeX: an excellent online introduction from David Wilkins.
Great reference book from the Indian TeX Users Group.
Excellent on-line tutorials from Andy Roberts.
More tutorials from the Indian TeX Users Group.
The TeX Catalogue for finding packages.
Getting help The first and best place to go is TeX.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/Q0e-2_b6diE" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/useful-latex-links/Statistical jokes
http://feedproxy.google.com/~r/Hyndsight/~3/OPTc4AQel8s/
Tue, 15 Jul 2003 01:44:47 +0000https://robjhyndman.com/hyndsight/statistical-jokes/Most of these jokes were posted to Usenet news groups. People who read such things collected them and put them on their web sites. I have shamelessly borrowed them, edited them and posted them here for the light relief of other statisticians.
The Biologist, the Statistician, the Mathematician, and the Computer Scientist A biologist, a statistician, a mathematician, and a computer scientist are on a photo-safari in Africa. They drive out into the savannah in their jeep, stop, and scour the horizon with their binoculars.<img src="http://feeds.feedburner.com/~r/Hyndsight/~4/OPTc4AQel8s" height="1" width="1" alt=""/>https://robjhyndman.com/hyndsight/statistical-jokes/