I'd really recommend this 'ultimate' .vimrc file if you're a Vim user...
I have forked it here...
I've also made a few very minor changes to open splits on the right and the bottom respectively and always include line numbers. I've also mapped vimdiff 'diff searches' from [c to [ and ]c to ], just for ease!
I'm doing some playing with the built in Veros examples and one of them is this amazing high res north Atlantic regional model...
The redder colours are warmer water and the bluer ones are colder water.
See the discussion paper on Veros for more information!
A few months ago I posted about using the Veros ocean model, which is written in 'pure Python'.
I have finally managed to find some time to get back to this and have produced a nice little gif (well, screen grab of one) of surface ocean circulation using the built in 4 degree global ocean example.
The animated gif isn't here because you can't put gifs larger than 10MB on the free Weebly version!
No scale, units or anything else at the moment but I'll get there eventually... ;)
I'll try and post more regularly from now on!
Last week I undertook software Carpentry instructor training. This is something that I have wanted to do for a long time and I was very lucky to be able to do this at Victoria University of Wellington in my home city.
Software Carpentry is a global movement to educate people in techniques that will help them create better software. Link here. There are also library and data Carpentry training workshops which are now collectively known as 'the carpentries', carpentries.org/. A real emphasis is on training irrespective of personal work area though and the idea that librarians, bioinformaticians, social scientists and people just like me can all get this training is really empowering.
Software Carpentry itself is broadly split into three strands; controlling a computer environment using the ‘shell’ (sometimes called the command line or terminal), scientific computer programming with Python or R, and version control.
The training itself was largely pedagogical (that is, learning to pass learning on to others) and this was quite new to me. We covered everything from working memory and cognitive load to live coding practice and how to organise a software carpentry workshop and everything in between.
All of the materials are available on GitHub here carpentries.github.io/instructor-training/.
For me what I found by far the most challenging, was preparing and the presenting a short teaching session which was (shock horror) filmed and then played back to you in small groups. I was amazed at the effectiveness of this approach since it was rapid fire and then repeated so that we could progress in light of what we personally thought of our performance whilst also taking the thoughts of our other group members into account. I chose the shell as my teaching vehicle and it was incredible how quickly I descended into some jargon even though I tried not to! I believe that I improved during the course but I will find out how successful I was when I have to do a live demo of this to other people in the software Carpentry community online as part of my post course assessment. Wish me luck!
As a side note, something that I hadn't used before was etherpad (etherpad.org/). This is a way of collaborating online in real time with others in a group where everyone's input is colour coded, editable and open access. I'd recommend you give it a go. I will certainly be using is again in the future.
An fantastic aspect of the software carpentry movement is the emphasis on the code of conduct., docs.carpentries.org/topic_folders/policies/code-of-conduct.html This ensures protection from discrimination and intimidation, whether conscious or otherwise in the instructor training and also in the courses which I will hopefully contribute to as a qualified instructor in the future. We all have aspects which make us different from the person metaphorically or physically next to us and so the knowledge that students and instructors are free to be themselves in a safe space is of comfort to me and I'm sure to many others also. I recently went to a conference where a similar code of conduct was expected of delegates and personally I found that just knowing that it was there made me feel empowered to be myself which boosted my confidence to be myself and be a better scientist and communicator.
Although I've been using Linux and shell based programming for about 14 years now, I never had any specific training in the style of software carpentry or actually anything remotely resembling it! I can confidently say that I would've programmed more efficiently, not deleted so many things accidentally, and saved myself and my fingernails a huge amount of pain via creating version controlled repositories and by writing better, more collaborative code!
I very much hope to qualify soon to be a fully fledged software Carpentry instructor soon so that I can pass whatever knowledge I have to others in the ever growing community that I'm proud to be a little part of.
Does it work?!
Ok let's try adding a photo I just took whilst waiting for the bus which never turned up... 🚌
.Here's some pics I took last month on my way to work of the Tangaroa (our flagship research vessel) returning to Wellington after her latest Antarctic voyage! You can find out more about the journey here... https://www.niwa.co.nz/our-science/voyages/antarctica-2018.
Just a very quick post about the versatile ocean simulation package, Veros. I've only just started using it but it seems very cool. I've already run some models on Linux and using Mac OSX and it almost entirely works out of the box.
More information is at the model's GitHub...
here's some screen grabs from there...
I'm looking forward to getting involved and have already been in touch with the lead developer regarding installation and its dependencies on the Python 'requests' package (http://docs.python-requests.org/en/master/). This has already resulted in a code change on GitHub! :)
I accidentally deleted a few things from my home directory today. I take, er, about 75% responsibility for this! Have a look at these instructions (from CTAN... ctan.org/mirrors/register?lang=en)...
, OK so the main point here is the `rsync --delete` option. I basically tried to put this in my home directory and... well, you can guess. It seems bizarre to me to suggest this as an option for such a major piece of software; why not GitHub or use some kind of svn server or something. Giving the user the ability (well, the suggestion) to delete things in their own space or, even worse, the in the root /var space seems at best dangerous and at worst irresponsible.
Anyway, like I said, I take most of the responsibility, but not all!
By the way I lost basically nothing because all my scripts were on GitHub and all my simulation configs are on code.metoffice.gov.uk. :)
Just a quick note to say that as of today we now have the Met Office UM consortium's latest atmospheric climate model running here in New Zealand on our new Cray supercomputer. The name it goes by is GA7.1 where the GA stands for Global Atmosphere. I'll give more information soon but I thought that this was a worthwhile line in the sand to shout about!
The science of this paper is discussed in the this article is Geoscientific Model Development Discussions...
This is a repost from the one that I wrote for the Deep South National Science Challenge (http://www.deepsouthchallenge.co.nz/) which funds my current research...
This was published in September 2017 but was actually written in June 2017.
Kia ora! Hello from New Zealand.
Sticking with one thing throughout my career has arguably been a bit of a problem for me but conversely I think this has actually stood me in good stead for my current job; for reasons I hope to explain.
I work at The National Institute of Water and Atmospheric Research (NIWA) in Wellington, the capital of New Zealand and my role is to develop, document, version control, etc, etc, the New Zealand Earth System Model or NZESM. Now I can already hear some of you shouting that New Zealand doesn’t have a big enough climate science community to develop an Earth System Model on its own and you’d be right. The NZESM is being developed alongside the UKESM (its UK parent model) with the aim of growing its own legs in the future. This is detailed in a lot more detail in our recent paper in Weather and Climate (see figure). This paradigm of several models using similar (if not identical) components is common nowadays; the UKESM and NZESM for example use the NEMO ocean model which is also used by other climate modelling groups in Europe. This need is well explained by the developers of the Norwegian Earth System Model, NorESM (http://www.geosci-model-dev.net/6/687/2013/).
“Despite the nationally coordinated effort, Norway has insufficient expertise and manpower to develop, test, verify and maintain a complete Earth System Model. For this reason, NorESM is based on the Community Climate System Model version 4 operated at the National Center for Atmospheric Research on behalf of the Community Climate System Model (CCSM)/Community Earth System Model (CESM) project of the University Corporation for Atmospheric Research.“
Anyway, what I’m trying to say in a roundabout way is that in a small research environment one has to be flexible and, in some ways, a Jack of all trades (I won’t finish the phrase but you get the idea).
This figure show the climate research landscape in New Zealand noting the contribution of the Deep South Science Challenge (http://www.deepsouthchallenge.co.nz) which funds the NZESM. The figure is from the Challenge's Research and Business Plan which is available here.
I started off my education when I left school as an accidental physicist. I was better at music and languages but I decided that I could do these later on or in my spare time to a half decent standard whereas physics… not so much. To be honest I began totally hating my compulsory C++ computing classes but after two years of realising that computers can be useful I decided to take a third year option in computational physics and even opted to do my Masters project in the simulation of electrical transport in disordered, soft matter semiconductors, i.e. solar (photovoltaic) cells and light emitting diodes.
After this I was working as a postdoc at the UK Defence Academy and came across a job doing climate model development at the Met Office. I’d heard from somewhere that they didn’t just employ specially trained meteorologists although at this point I didn’t quite believe it. I got an interview and got the job. I spent the next two years working in climate model development and evaluation with a particular emphasis on clouds and radiation. The training I received was first class and I now very much consider myself a meteorologist and a physicist. This was undoubtedly a hugely formative experience for me; lifting me hook, line and sinker out of my comfort zone and really enjoying the challenge.
After those two years I jumped ship into environmental consultancy. This new job could hardly have been more different to the last and I again had to learn quickly but this time about waste management, planning law and the environmental impacts of recycling and reuse and so on. It was a challenging role that’s for sure but at this point I started to realise that perhaps I wasn’t cut out to do the same thing all the time and that this wasn’t a bad thing.
My next foray into the relative unknown led me into paleoclimate research working as an industrially sponsored postdoc at Bristol University. This was a role which in some ways was very similar to my role at The Met Office but in other ways super different. Similarities included the fundamental philosophy of model development and the evaluation of these models against available observations and reanalyses. However, since most of the models that I was helping to develop were designed to be used on ‘deep time’ continental configurations, i.e. well over 100 million years ago, one also has to consider ‘proxies’ for validation. This becomes obvious when you consider that the dinosaurs didn’t have thermometers and barometers to measure the weather and therefore we have to rely on proxy measures such as temperature-dependent isotope ratios and geographical distributions of animals and plants with modern day analogues. Much of our model development took place on modern day configurations of our model however to enable calibration to known climate states with good observational coverage.
Round about the time that my postdoc was coming to its natural end, with the natural crazed paper and proposal writing, I came across a job advert for an Earth System modelling position in New Zealand. My boyfriend and I had discussed working abroad at some point in our lives and it seemed like a fun thing to at least have a go at applying for. After a good 6 months of applying, interviewing, serving 3 months’ notice, sorting visas and travelling for 35 hours I arrived in Wellington in December 2015.
My job here at NIWA is highly varied and challenging but hugely rewarding when things work! We have a growing team of users of the nascent NZESM across New Zealand and it's part of my role to support individual users’ research needs whilst making sure that we are both keeping apace with developments from the Met Office led Unified Model (UM) consortium and contributing our own scientific advances back in the other direction. This figure shows the location of the UM consortium partners (from http://www.metoffice.gov.uk/research/collaboration/um-partnership). More information on the Met Office Unified Model can be found here http://www.metoffice.gov.uk/research/modelling-systems/unified-model .
If I had to say what is the one thing that makes my life as a developer here easier, it is definitely the use of web based code repositories. We make use of two main types in our work; the Git-based GitHub for technical infrastructure (e.g. Rose, FCM and Cylc) and the Apache Subversion-based Met Office Science Repository Service (MOSRS) and NEMO ocean model repository in Paris.
The beauty of a code store like this is that it makes collaboration a doddle and, when used properly, simplifies the management, development and version control of whatever code is being tracked. Indeed it is not only ‘science’ code which we track in this way but also support and upgrade ticketing, project management timelines and wiki pages, all version controlled and available instantly to all registered users worldwide.
A concrete example of how the New Zealand community has been using this collaborative capability and contributing to model development is in the atmospheric chemistry code. A few years ago, advances were made here but at the time there was no practical way of communicating these changes back into the trunk (the central master branch if you will) of the Unified Model code. Now that MOSRS is available to us, our researchers have been able to make a branch to the UM trunk which can be trivially included in development suites by any user with access to the repository and eventually committed into the UM trunk for all to use.
A further example of the utility of the service concerns site portability of code configurations. This is very important when one considers that the same suites of code need to be run by research groups which are not only geographically spread but also use different HPC architectures. The global atmosphere research which happens in the UM consortium advances in documented in integer steps and the most recently documented version is GA6 (http://www.geosci-model-dev.net/10/1487/2017/). One of my first tasks in my current role was to port the next version, GA7, to our IBM Power6 HPC from a configuration which was originally designed to run on a Cray XC40 machine. It’s beyond the scope of this post to give all the gory technical details of this but in short it was necessary to change compilers, job submission methods, boundary condition file locations and so forth. By doing this we are able to provide diversity to the results of the GA7 configuration by e.g. comparing compiler versions as well as improve our own national capability in climate research by providing boundary lateral boundary conditions to downscaled regional climate modelling studies over the New Zealand region. An example of previous regional climate research output can be seen in this figure (from https://www.niwa.co.nz/climate/research-projects/regional-modelling-of-new-zealand-climate).
Since moving to New Zealand the ability to work remotely has gone from being a convenience once in a while if I needed to work at home to let a plumber in or something to being an everyday essential part of my workflow. Pretty much all of my work now relies on site-, device-, and operating system-independence. Even for the example of writing this blog post I have already used two devices in two locations and will very likely use at least one more of each before I’m done. As well as these ‘expected’ benefits, next week I will also be able to video conference in to the Met Office for the annual UM users’ workshop. I had planned to be there in person but my health had other plans! There is the slight disadvantage of being in a wildly different time zone but that’s something which is not going to change.
New Zealand has considerable expertise in atmospheric chemistry and high latitude climate processes and so it is fitting that we have been tasked with contributing to CMIP6 simulations by providing ozone boundary conditions to other members of the UM consortium. Therefore although we are a small community here we do not feel isolated from the rest of the consortium and indeed contribute a diversity of knowledge and skills whilst building national capability here in the South Pacific. This capability is crucial to the work of the Deep South National Science Challenge which funds the NZESM model development. This Challenge also funds process- and observation-based research into the Southern Ocean and Antarctic regions, the impacts and implications of climate change on New Zealanders (and in particular the Māori community through the Challenge’s Vision Mātauranga programme) as well as a wide-ranging engagement remit.
Now I thought it might be useful to others to share some of the other things that I have learnt in my current job which I have found useful and which I hope may prove useful to others also.
These are a way of displaying code, comments, markup text and figures within a webpage in a standard internet browser. Not only does this make for easier development (I find it a lot easier to see what I’m doing all in one place personally) it also means that the notebooks themselves can be shared as single files containing as much information as you want, including figures, so that the person you’re sending it to doesn’t have to rerun the code or deal with multiple attachments. I personally use Python as my coding language in Jupyter notebooks but you can also use other languages too, such as R and Ruby. This example figure comes from the Project Jupyter webpage, www.jupyter.org.
At the annual New Zealand eResearch conference in 2016 (http://www.eresearchnzconference.org.nz) I first came across the idea of Hacky Hours. These normally consist of one hour a week where people can get together to (very) informally discuss anything that they want regarding their work on or with computers. The figure on the left shows an example whiteboard at the end of one of these sessions! Sometimes I have been the only one able to attend and I’ve simply gotten on with my normal work with a change of scene (highly recommended!) but most of the time there are about 5 of us. We’ve had presentations on the R language, baffled ourselves with TED talks on statistics, had a live demonstration of cloud computing servers, learnt about quantum computing and seen first hand how useful Git and GitHub can be when collaborating on a coding project. All in all I would really recommend you give Hacky Hours a go; they involve essentially no planning and have proven to be very useful here at NIWA.
I joined Twitter (www.twitter.com/jonnyhtw) in November 2014 and it has been a very positive experience for me. As well as keeping informed about all kinds of environmental issues (and a lot else in between!), as a direct result of Twitter I have been involved with an online mission to try and end the use of the ubiquitous rainbow colour scale in climate science, partly motivated by my own fairly rare form of colour blindness. This mission has the hashtag #endrainbow and has even resulted in a short Communication in Nature. This figure is from https://twitter.com/ed_hawkins/status/808645275082489856 relating to a previous Climate Lab Book post, http://www.climate-lab-book.ac.uk/2014/end-of-the-rainbow/.
In addition to #endrainbow, I should also mention that the post you are reading right now is a direct result of interactions via Twitter.
Software carpentry and ResBaz
I was lucky enough to attend the Research Bazaar, or ResBaz, at Victoria University of Wellington in 2016. info on the global 2017 ResBaz can be found here https://2017.resbaz.com and a photograph of mine from the event can be seen on the left). In the words of the ResBaz website (I couldn’t say it better myself) ‘The Research Bazaar is a worldwide festival promoting the digital literacy emerging at the center of modern research.’ This was a great opportunity to get involved with eResearch in my new home city but away from my normal 9-5. One thing that I definitely didn’t expect to get involved with was constructing a small website on Wellington author Katherine Mansfield http://jonnyhtw.weebly.com/resbaz-2016.html as part of the ‘Data Olympics’! This ResBaz was also the first time I had come into direct contact with a course teaching Software Carpentry, which is a global movement teaching digital skills to researchers (https://software-carpentry.org). I really wish that I had had the opportunity to attend one of these courses when I started my PhD and would encourage others to attend if they are able! Excitingly I hope to be helping out as one of the teaching assistants in my first Software Carpentry course soon.
In summary, my work as an Earth System model developer is enabled by collaborative, mainly open source, cloud based tools, without which my work would be essentially impossible. I am forever striving to make things even more portable, flexible and user friendly for myself and for other climate and Earth System modellers in New Zealand. In my opinion this makes things easier for everyone and increases transparency, traceability and safety (e.g. through automated version control).
I also hope that I’ve convinced you that having a non-standard career path can work out OK. Early career scientists nowadays are under huge pressure to publish in high-impact journals, write successful proposals, have their career planned down to the Nth degree by the time they’re 21 and generally be superhuman. My advice would be to never work in isolation, ask for help and use open source and version control tools wherever possible. Thinking about it, all three of those suggestions are basically different facets of communication. As someone much wiser than me once told me (and I’m paraphrasing because I can’t remember exactly who it was!): ”If you don’t communicate your research to others, then you might as well not have done it!”.
I hope that this has been interesting and maybe even useful to you. Please feel free to stay in contact! I’ll leave you with a picture of our snazzy new Cray XC50 supercomputer which will be installed later this year! You can read more about it and its sibling machines here.
Haere rā, goodbye and thanks for reading!