2015.23..25 Catching Up Is Hard To Do

The past weeks we’ve had no Perl 6 weekly. Trying to catch up with so many things happening on-line and IRL is quite a lot of work and prone to omissions. Please bear with us while we get things on track again.

Blog Posts

For those of you who missed them, there were some excellent blog posts in the past 3 weeks (in chronological order):

Conferences, Conferences, Conferences

The conference season has really started again. Some Perl 6 related videos from the German Perl Workshop in Dresden:

Some Perl 6 related videos from OSDC.no in Oslo, Norway:

And some Perl 6 related videos from the YAPC::NA in Salt Lake City, Utah:

And some related blog posts:

The GLR Is Spreading Its Wings

Patrick Michaud has started a new version of the List And Iteration Synopsis. If you want to keep abreast with the developments there, please check it out. If you have any comments, let them be heard now rather than later!

Rakudo 2015.06 Was Released

Rob Hoelz made the Salt Lake release happen, which came in at 200+ commits. Since then we have seen about 40 commits for the next release, mostly consisting of bug fixes, performance enhancements and the odd new deprecation.

Busy FROGGS

Tobias Leich has been busy. He made:

  • Rakudo on MoarVM buildable on Windows with gcc/gmake
  • Use run()/shell() as a replacement for pipe()
  • Added libffi support, which should let build MoarVM on all platforms

This should make MoarVM build on all Debian platforms except Hurd.

Unicode-8 Support Is Now In

As of 23 June, Rakudo on MoarVM supports Unicode 8.0, only about a week after its release! Appropriately for the Perl 6 Party, it has a code point for Bottle With Popping Cork!

Covariance And Contravariance

A Fresh Look At An Old Issue

Giuseppe Castagna has written an interesting article about (static) typing for Perl 6:

The target reader of (the first part of) the article is every Perl 6 programmer, which is why all the above is explained by examples of Perl 6 code (no formula, theory, theorem, or property whatsoever) and I tried to do my best to make the article completely self-contained…. The second part of the paper targets language implementers and designers.

I haven’t really gotten around to reading it thoroughly, but I find it very interesting that all examples in this article are using Perl 6! Giuseppe is welcoming comments on the per6-language mailing list.

Winding Down

Perhaps this Perl 6 Weekly was a bit too sparse on events on the #perl6 channel. Trying to catch up with 3 weeks of backlog has become such a daunting task, that I have only scanned the first week of June for interesting bits (so far). If there are any other interesting bits in the backlog in weeks 24/25, I will mention them in the weekly of week 26.

If you happen to be in Karlsruhe this weekend, find me at the GPN15!

Hello, dear esteemed readers!

I’ll be attending the Gulaschprogrammiernacht 15 in Karlsruhe this weekend (starting this Thursday evening already), so if you’re interested to talk about Perl 6 or maybe even write or port a library or something, I’ll gladly give you a portion of my time :)

If you’re in Karlsruhe and are not attending, that’s no problem, as there’s no entrance fee, so you can just teleport on by and see what’s up. You can find out more about the event at http://gulas.ch/.

If you want to meet up, you should be able to reach me on the freenode IRC network on #perl6, which I’ll likely check regularly.

See you there, or maybe not!
– timotimo

2015.22: While async & conc are still a bit crashy, enjoy these unrelated improvements …

Last week’s blog post by masak about the “Send More Money” mathematical puzzle kind of sent me off on a little coding spree. A quick profiling run of the “loops” version of the solution quickly showed that a sizable chunk of the run time was spent inside the implementation of the “next” keyword (well, sub really.).

After just a bit of fiddling around with the implementation of “next” – which had been pessimized some time ago for some reason – together with lizmat, the benchmark went from 22s to 15s, about a 30% improvement. This is not only purely less time spent running, but also less work done GCing.

Not only “next”, but also “redo” and “last” have been improved in the same way.

lizmat also went ahead and added a :as parameter to the categorize method like the classify method already had. This lets you change what the value that came in will end up as in the resulting hash. On top of that, the classify and categorize subs also work with a :into named parameter, which allows you to supply the exact type or instance to put the values into.

hoelzro put on his grammar hacking robe and NQP hat and improved error messages for cases when constructs get opened, but not closed. See bugs #125247 and #125153 on the RT.

Another thing hoelzro made work was the .push method for the Buf class (buffers that contain integers of a fixed size, closely packed together).

And then there’s also an interesting fix where an array of just a single pair in adverb notation would have the parser attempt to parse it as a reduce operation. Here, look:

[+] 1, 2, 3   # reduce operation
[:a]          # an array with a single Pair in it

This used to blow up, but works now.

Something I did was dump more information about logging and guards in the spesh dump log. I also found a bunch of places where creating P6int instances didn’t go through our integer cache that hopes to reduce the number of different incarnations of numbers 0 through 15 (now -1 through 14).

Finally, here’s a nice short blog post from domm about his experience porting a very simple script from perl 5 to Perl 6.

Looking forward to what the next week will bring :) In any case, I hope yours will be fine :)

2015.20&21: Blogs, Grant, Release and Progress

The past weeks we’ve seen quite a few nice Perl 6 related blog posts. Just in case you haven’t seen them, I would like to repeat them here for you:

The Annotated Berlin Consensus

At the 2015 QAH in Berlin, a consensus was reached on a number of Perl toolchain issues, with a particular focus on governance and recommended standards of care for CPAN authors. The official version in Markdown can be found on Github. David Golden also blogged an annotated version of it. Although one could argue this is not strictly Perl 6 related, all of the issues mentioned in the consensus, will come to the Perl 6 ecosystem sooner or later.

Advancing the MoarVM JIT

This grant proposal by Bart Wiegmans (brrt) has been accepted. This is great news, can’t wait for bare Rakudo startup time to drop below 0.05 seconds!

Bare startup time below 0.1 second

In the past week, the bare startup of Rakudo on MoarVM finally dropped below 100 milliseconds on most developers’ machines. This is now only a factor of 20 slower than Perl 5. However, I’m told that starting Perl 5 with Moose (which from a feature point of view, gets it closer to Perl 6), is now slower than starting Rakudo!

Dresden

We also had the 2015.05 compiler release of Rakudo, named Dresden (after the recent German Perl Workshop there). The list of changes is quite extensive, coming in at 370 commits since the last release.

Changes since the Dresden release

  • use trace: lexical pragma to trace program execution on STDERR
  • Empty IO::Path / “”.IO no longer allowed, it used to be equivalent to “.”

GLR underway, watch your modules!

One can say that the GLR (Great List Refactor) is now really underway. Several modules in the ecosystem are now known to fail because of GLR induced changes. If you have a module in the Perl 6 ecosystem, it would be nice if you could check it for GLR induced bitrot using the latest Rakudo!

Shutting down

This Perl 6 Weekly came in a little late. Hopefully, future Perl 6 Weeklies will come in a more timely manner. Until then, keep up the good work!

2015.19: Events & Conferences

During the last week, there were multiple conferences where perl6ers could be found:

  • Larry Wall gave a talk at Collision in downtown Las Vegas and a Q&A session afterwards using twitter and an app called pundit. I have no idea if there’ll be recording of the talk, but supposedly, you’d be able to get at the Q&A stuff somehow somewhere …
  • At the German Perl Workshop froggs talked about native library bindings, using libxml2 as an example.
  • Also at the GPW, lichtkind gave a talk about functional programming in perl6, but the slides are in german.
  • Another talk from the GPW is rurban’s overview over VMs related to perl. The talk and the slides are in english.
  • At the OSDC.no there were many perl 6 related events, which can all be found on the “list of talks tagged Perl 6“. Sadly, no links to slides can be found there, and I don’t yet know about recordings.

But there was also some work being done – there were post- and/or pre-conference hackathons, too.

  • froggs threw pretty much all files that were in our roast repository into rakudo’s spectest.data, which means that a whole bunch of test files weren’t actually being run until now. Those are all fudged properly now, though, so that we can further decide to throw out tests that haven’t stood the test of time, for example. [correction: actually a bunch of those are still not being run, but at least got triaged]
  • TimToady added line number annotations to warnings more often, and also hunted down a bunch of “leaking” Failures, which means a Failure gets created at some point, but never sunk or inspected. (remember, Failures are basically unthrown exceptions)
  • lizmat has improved @*INC and the “use cur” pragma (which is going to replace “use lib” when it works as wanted – of course it’ll also get the “use lib” name at that point) a whole lot.
  • TimToady is moving forward to make Nil no longer disappear when doing list flattening or assignment. Right now, there’s an Empty value that’ll do the “disappears in flattening” and Nil no longer disappears in lists. If you yourself want this behavior, please use () instead, as Empty is most probably going to disappear once the “switch-a-roo” is done :)
  • thanks to raydiak “undeclared variable” errors now show a more correct error position and don’t give you the list of what kinds of rules/tokens the parser was expecting at the point of failure
  • lizmat restored the performance of IO::Handle.words and .lines when providing the :eager parameter, that you can use to not pay the “lazyness cost”. Of course, this is what we want the GLR to “automatically” provide in most cases: not paying the lazyness cost if you’re actually eagerly using something that can be lazy if it wants to (or if you want to).
  • jonathanstowe improved the “earliest” block which now handles having both “more $foo” and “more *” blocks in it.
  • brrt gave me the go-ahead to merge the “jit_devirtualize_reprops” branch, which turns indirect calls to a bunch of operators on objects into direct calls (if the type is known at jit-time). This is also something we’d have wanted for brrt’s sponsored work on improving the MoarVM jit further.
  • froggs is quite far on the way towards getting “ignoremark” support into rakudo, which lets a regex (or parts thereof) match regardless of any combining characters applied to the target string. [correction: in fact, froggs is so far along the way that ignoremark is already in rakudo!]
  • jonathanstowe added some more concurrency related docs to our documentation repository (and website).
  • moritz described how to get default values at object creation time into your classes in the classtut

That’s all I have for now. There wasn’t quite as much progress with so many perl6ers traveling and attending or giving talks and workshops.

And at 666 words (not including this paragraph), I feel this is a good place to stop writing. Have a nice week, everybody!

2015.18: Startup time rectified, hashes unordified, startup speedupified, puns deduplified

This week pleases me greatly, because there’s performance improvements to report!

Startup time & performance

lizmat worked hard to make initialization of the module installation database in rakudo lazy, so that any script that doesn’t “use” any module won’t take the performance hit. This (together with a bunch of other improvements) gets startup time (as measured by compiling and running an empty program) back to 0.17s on my machine (previously it was 0.31s) and the memory usage went from about 85 megs to 69 megs (these numbers come from the time command outputting the maximum resident set size). On lizmat’s machine, that reduced the time it took to do a full spectest run from 1620 to 1435 CPU seconds.

On the other side of the module installation database coin, froggs and jnthn just had a little breakthrough that will finally allow the json database to be replaced with a MoarVM serialization blob. Needless to say, this will be a whole bunch faster to load and store when we actually do need to load the database.

On top of these big improvements there’s of course been small improvements again. After profiling found out that a lot of time is spent decoding utf8 during startup, jnthn switched a bunch of things that are known to always be valid latin1 to decode that instead. This reduces the instructions run during startup by about 10%.

Another nice change is that .moarvm files used to contain not only a string heap section but also a chunk of code that used to create a list of all these strings. Now the string heap is used for both purposes, shaving a good 60 kilobytes off the CORE.setting.moarvm (still about 9 megabytes big) and reducing the startup time by about 2%. And then there’s a few sundry changes that all shave off a few percent here and there.

Adding to this heap of performance improvements all around, hashes on MoarVM no longer preserve insertion order. This feature – which users were discouraged from relying on anyway – was implemented in uthash by a doubly-linked list. jnthn removed that doubly linked list and made hashes save a noticable bit of memory: 16 bytes per hash entry and another 8 bytes per hash.

In the future, uthash will be more heavily modified so that strings representing the same text in different storage formats (8 byte per character vs full NFG) will result in the same hash. Then we’ll finally be able to store simpler strings more compactly. I’ve been looking forward to this for a long time, and now it seems to be within reach :)

Something I forgot to mention last week is that a whole bunch of methods on Str are now much faster because they have gotten their own implementation rather than relying on Cool to implement them (and do a needless coercion first in every case). This change was inspired by making .chars much faster in the very same way.

Bug Fixes

There used to be a bug where you’d get maddening errors like this:

Calling rl_get_screen_size(TypedPointer[Int], TypedPointer[Int]) will never work with declared signature (TypedPointer[Int], TypedPointer[Int])

This came from NativeCall using the “make_pun” meta-object method that erroneously created a fresh, distinct class each time TypedPointer[Something] was used somewhere. That method call has now been replaced with just “pun”, which goes through a cache first.

ptc found and fixed a few small memory leaks in MoarVM.

Today jnthn spent time on known multithreading-related instabilities and oddnesses. A big part of that was fixing a race in the Fixed Size Allocator which is used for frames, on-stack-replacement, stuff like that. There was a CAS-based lock that suffered from an ABA problem, which basically means one thread interpreted “two changes” (there and back) as “no change” (“go ahead”) and the list of free spots in the allocation pool could get corrupted, leading to memory corruption in objects when two threads use the same spot for different types of objects. The multi dispatch cache also just got a lock added to it, too. There is still more waiting to be found and fixed, though.

I personally got a tiny bit of work in, as well: If you ever encountered the error “Can’t adverb that!”, you’ll now find that it’ll also give a little hint as to what exactly the parser thought you were trying to adverb (because the precedence level of adverbs takes a little getting used to, IMO).

There used to be a problem related to statement modifiers and the given and for keywords where the value of $_ would get messed up, and block-less “gather” sometimes didn’t work where just adding curly braces would fix the problem. Thanks to jnthn, that’s fixed now as well.

Implemented features

Thanks to lizmat variables can now take “will” blocks, which correspond to phasers. Those already implemented are enter, leave, keep, undo, first, next, last, pre and end. Also, unimplemented “will” types now throw an error instead of failing silently. In addition to that, “lazy attributes” are now supported in rakudo. These let you write code like this:

class Example {
    has $.attr will lazy {
        # do some expensive set-up here
        "hello" # value returned from this block will be used
    }
}

When the attr is first accessed, it will run the code to set up the value and after that it will behave like any regular attribute.

After making List.roll faster last week, lizmat also made List.roll(*) (and List.roll(Inf)) return a list that knows it’s infinite.

If you set the environment variable RAKUDO_VERBOSE_STACKFRAME to a number, that number of lines of code around a line indicated in a stack trace will be shown. Another change to stack traces is that instead of a few select routines from the core setting, all routines in the core setting will now be hidden from backtraces by default – you can get full backtraces with the –ll-exception flag as before.

Even though this isn’t strictly a feature, there’s also been more changes regarding the GLR – mostly how flattening and nonflattening happens in relation to “for” and “map” in both their sub and method form. Lists and Parcel have been moving closer and closer and soon the Parcel class will be thrown out.

Ecosystem goodness and documentation

Even though recent changes to the module installation system and making hashes unordered brought a few bugs and problems out of hiding, there’s more nice stuff to be found in the ecosystem now!

  • Module::Minter gets your own module started with the basic files every such project needs (by dnmfarrell)
  • Template::Protone lets you write templates that compile down to fast perl6 subs and let you easily embed perl6 code (by tony-o)
  • ANTLR4::Grammar lets you parse an ANTLR4 grammar and use it to parse stuff. (Though it’s not been put into the ecosystem yet, I decided to highlight it here none the less.) (by DrForr)

ptc has pushed a whole bunch of commits to the perl6 examples site and ptc, stmuk, jonathanstowe and moritz have been adding and modifying more and more stuff on the perl6 documentation site.

labster wrote a blog post “Three Tales of Second System Syndrome” that he put up on his site, hoping to get a bit of feedback before spreading it further. However, it reached Hacker News only shortly after he went to bed and there’s now 121 comments in the corresponding thread.

That’s it for today!

Very strange; even though it feels like I’ve written the longest post in a few weeks, I also have a sneaking suspicion that I’ve forgotten to mention multiple things!

In any case, it’s time for me to finally finish up and maybe get back to annotating MoarVM’s code with a few hints for Valgrind and its tools. Have a great week!

2015.16 + 17: A release, NFG and more GLR work

Hello everybody! I’m back after a postless week. Since there was a compiler release in the earlier of the two weeks, I’ll highlight some of the things this compiler release brings and advances and changes made since then.

Sadly, there’s currently a major startup performance regression that has to do with the way we install modules and their files into a database-like format. This will most likely mean there’ll not be a Rakudo Star release this month.

Pragmatism

Pragmas (like “use soft” or “no strict” or “use fatal”) now properly work with lexical scoping.

Along with proper scoping of pragmas, “use fatal” now has a proper implementation. What that implies is for example “try 1/0″ immediately blowing up rather than appearing to succeed and returning a Failure object.

The third thing I can mention is that -Mstrict now works to make programs passed on the commandline with -e strict. In fact, -M works with all pragmas now.

Performance improvements

One of the first big improvements came from nwc10 going over MoarVM’s serialization format and kicking out a whole bunch of unnecessary 0 bytes all over the place and also making the deserialization step take fewer CPU cycles in general. This leads to noticably smaller .moarvm files, for example, but also slices off a few cpu cycles during startup. We’re still quite a bit away from getting down to 0.01s startup, compilation and execution for an empty program, but it seems we’re getting closer and closer each month.

By not trying to hold the complete output in RAM, MoarVM’s profiler is now able to spit out json blobs so gigantic that not even the profiler data analysis program written in C++ to handle more data will open it. Before that change (many thanks to flussence for this, btw!) MoarVM would very quickly balloon up in its memory usage and get OOM killed.

lizmat built a eager variant of List.pick(*) that’ll give you a shuffled version of a list faster than the lazy variant. However, List.pick and List.pick(*) themselves got 3% and 7% faster. Something else that’s gotten faster thanks to lizmat is the clone method for the cases where no changes are made to the resulting object via named parameters.

Documentation

On the documentation side, doc.perl6.org received a 857 inserted lines over the two week period covered. Thanks to moritz, ptc, stmuk, skids and jonathanstowe for their contributions! The files that got changed the most were concurrency, control and objects in the language section. If you miss something in the doc repo or the website, feel free to ask on the IRC channel, open an issue or send a pull request for the WANTED file. Of course if you like, you can also send pull requests for changes or additions to the docs directly.

moritz published a blog post about writing docs on his blog that I found interesting.

And then there’s examples.perl6.org which has seen many, many commits from ptc and stmuk. Kudos!

NFG, GLR, other cool bits

I’d write about the NFG things jnthn has implemented, but he already blogged about it on his personal blog and explained a lot, so I’ll just send you over there to read. If you’re running rakudo from the git repository, you’ll have the Str class backed by NFG. If you’re running the latest compiler release, you’ll be able to use .NFG on a Uni object, but NFG isn’t the default yet.

On the GLR side of things, TimToady has been working more on handling things that flatten or stay itemized. It’s hard to turn the changes and discussion into a succint summary, though. One thing that’s gotten noticably more consistent is the sequence operator (“…”). The latest discussion was on how the two forms of looping over an array should be called: One that treats a list as a collection of items and one that causes a list to flatten its inner lists and iterate over the result.

The rotor method of List and Supply now take a list of pairs as their arguments in the format Take-How-Many => Move-How-Many-More. For now, if you supply two integers, you’ll get the old format of Take-How-Many, Step-Back-How-Many [correction: the backwards compatibility mentioned in this sentence only works in the released version]. This makes it more sensible (the previous default value for Step-Back-How-Many was 1, which is sort of unexpected).

psch and hoelzro have both invested time in the REPL. psch made the tr/// operator aware of adverbs, and of course psch has spent more time on the JavaInterop side of rakudo.

You can now assign a finite list to an infinite slice of an array or assign an infinite list into a finite slice of a list, like this:

my @the-list;
@the-list[15..*]  = 1, 0, 1, 0, 1, 0;
@the-list[0..10] = 0, 2 ... *;
say @the-list;
#-> 0 2 4 6 8 10 12 14 16 18 20 (Any) (Any) (Any) (Any) 1 0 1 0 1 0

As for froggs, have a look at this bit of code he made work just today (well, yesterday):

> say (0..0xFFF0).chrs.join('') ~~ m:g/<:NumericValue({+$_ > 0})>+/
「123456789」 「²³」 「¹」 「¼½¾」 「١٢٣٤٥٦٧٨٩」 「۱۲۳۴۵۶۷۸۹」 「߁߂߃߄߅߆߇߈߉」 「१२३४५६७८९」 「১২৩৪৫৬৭৮৯」 「৴৵৶৷৸৹」 「੧੨੩੪੫੬੭੮੯」 「૧૨૩૪૫૬૭૮૯」 「୧୨୩୪୫୬୭୮୯」 「୲୳୴୵୶୷」 「௧௨௩௪௫௬௭௮௯௰௱௲」 「౧౨౩౪౫౬౭౮౯」 「౹౺౻౼౽౾」 「೧೨೩೪೫೬೭೮೯」 「൧൨൩൪൫൬൭൮൯൰൱൲൳൴൵」 「෧෨෩෪෫෬෭෮෯」 「๑๒๓๔๕๖๗๘๙」 「໑໒໓໔໕໖໗໘໙」 「༡༢༣༤༥༦༧༨༩༪༫༬༭༮༯༰༱༲」 「၁၂၃၄၅၆၇၈၉」 「႑႒႓႔႕႖႗႘႙」 「፩፪፫፬፭፮፯፰፱፲፳፴፵፶፷፸፹፺፻፼」 「ᛮᛯᛰ」 「១២៣៤៥៦៧៨៩」 「៱៲៳៴៵៶៷៸៹」 「᠑᠒᠓᠔᠕᠖᠗᠘᠙」 「᥇᥈᥉᥊᥋᥌᥍᥎᥏」 「᧑᧒᧓᧔᧕᧖᧗᧘᧙᧚」 「᪁᪂᪃᪄᪅᪆᪇᪈᪉」 「᪑᪒᪓᪔᪕᪖᪗᪘᪙」 「᭑᭒᭓᭔᭕᭖᭗᭘᭙」 「᮱᮲᮳᮴᮵᮶᮷᮸᮹」 「᱁᱂᱃᱄᱅᱆᱇᱈᱉」 「᱑᱒᱓᱔᱕᱖᱗᱘᱙」 「⁴⁵⁶⁷⁸⁹」 「₁₂₃₄₅₆₇₈₉」 「⅐⅑⅒⅓⅔⅕⅖⅗⅘⅙⅚⅛⅜⅝⅞⅟ⅠⅡⅢⅣⅤⅥⅦⅧⅨⅩⅪⅫⅬⅭⅮⅯⅰⅱⅲⅳⅴⅵⅶⅷⅸⅹⅺⅻⅼⅽⅾⅿↀↁↂ」 「ↅↆↇↈ」 「①②③④⑤⑥⑦⑧⑨⑩⑪⑫⑬⑭⑮⑯⑰⑱⑲⑳⑴⑵⑶⑷⑸⑹⑺⑻⑼⑽⑾⑿⒀⒁⒂⒃⒄⒅⒆⒇⒈⒉⒊⒋⒌⒍⒎⒏⒐⒑⒒⒓⒔⒕⒖⒗⒘⒙⒚⒛」 「⓫⓬⓭⓮⓯⓰⓱⓲⓳⓴⓵⓶⓷⓸⓹⓺⓻⓼⓽⓾」 「❶❷❸❹❺❻❼❽❾❿➀➁➂➃➄➅➆➇➈➉➊➋➌➍➎➏➐➑➒➓」 「⳽」 「〡〢〣〤〥〦〧〨〩」 「〸〹〺」 「㆒㆓㆔㆕」 「㈠㈡㈢㈣㈤㈥㈦㈧㈨㈩」 「㉈㉉㉊㉋㉌㉍㉎㉏」 「㉑㉒㉓㉔㉕㉖㉗㉘㉙㉚㉛㉜㉝㉞㉟」 「㊀㊁㊂㊃㊄㊅㊆㊇㊈㊉」 「㊱㊲㊳㊴㊵㊶㊷㊸㊹㊺㊻㊼㊽㊾㊿」 「㐅」 「㒃」 「㠪」 「㭍」 「一」 「七」 「万」 「三」 「九」 「二」 「五」 「亖」 「亿什」 「仟」 「仨」 「伍」 「佰」 「億」 「兆」 「兩」 「八」 「六」 「十」 「千卄卅」 「卌」 「叁参參叄」 「四」 「壱」 「壹」 「幺」 「廾廿」 「弌弍弎」 「弐」 「拾」 「捌」 「柒」 「漆」 「玖」 「百」 「肆」 「萬」 「貮」 「貳」 「贰」 「阡」 「陆」 ...

As you can see, you can now match against a character’s unicode property value inside a regex. In this case we’re looking for unicode characters that have a numerical value that’s bigger than 0. Pretty cool, IMO!

On the ecosystem, there’s now a module that lets you write a grammar definition in BNF form and use it just like a Perl6 Grammar (that’s because it actually generates a Perl6 Grammar class for you). It’s made by tadzik and you can have a look at one of its test files here. There’s also a POD parser (for the perl5 variant of POD) by dnmfarell on github and the ecosystem.

The last thing before I finally go to bed will be the release itself. We have masak to thank for the release process itself, but I was very glad to see so many different people work on so many different pieces of the whole. I may only be making this up, but it feels like we’ve gotten more contributors :)

Well, at least the weekly is pretty big this time ’round, and I don’t think that’s because it covers twice as much time as usual, and there’s already more cool things to look forward to in the coming week :)

So you all have a great week now!

2015.15: We figured out the truth about “0”!

Jonathan Workington

jnthn just published a new blog post last night describing the fund that was recently set up for Perl 6 development (thank you, WenZPerl!) that’ll be used among other things for funding more jnthn time until the release coming in december. There’s also a lot of information on what jnthn has recently been doing in that blog post:

  • Unicode database in MoarVM updated to version 7.0
  • unsigned types in NativeCall (already mentioned in the previous p6weekly)
  • a whole lot of RT tickets fixed, like improved error messages for BEGIN-time things and constants, “where” constraints on attributes and variables
  • the beginning of the Uni class and its features, like different normalization forms (as specified by Unicode)

Performance improvements

Nicholas Clark has been churning out patches to reduce the number of cpu instructions “used up” during rakudo’s startup and also improved the efficiency of our serialization format.

TimToady improved the MATCH method in both rakudo and nqp that are responsible for creating the high-level object that gets created for a regexp match. That’s quite a hot method and initial, noisy measurements showed it shaves off about 5% of the time spent in the “stage parse” of rakudo’s core setting.

In addition to MATCH being faster now, rakudo (and nqp) used to flip ginormous amounts of string data whenever it encountered a <?after …>, but now [correction:] if the length of the pattern inside is known, it turns into a <?before …> with the pattern itself flipped.

I’ve built support for a whole bunch of reprops (push/pop, elems, access to attributes, keys or positions, …) into the JIT for the cases where we couldn’t generate a super simple piece of assembly code to handle something (like pushing into a list that may have to enlarge the list) but we still know what exact function we’ll be calling. Unfortunately my desktop is currently not fully assembled, otherwise i’d have some benchmark data for you.

labster gave Str its own “chars” method (as opposed to inheriting it from Cool), which makes .chars on a String work about 40% faster. Wow!

Semantic improvements

Something that may have surprised a lot of people (myself included) was that Perl 6 claims that the truth value of “0” ought to be False. That is no longer the case, and so the rule for the truth value of strings is now just “does it have any characters in it?”. I think that’s more sensible.

TimToady improved the parser’s handling of heredocs in constant expressions and BEGIN blocks. There’s a bit of cheating necessary in the parser to make all of this work, but masak wrote a post on his blog about this very topic that I recommend you read.

TimToady also implemented Complex literals inside < >, which means that <3+2i>.WHAT now properly results in a Complex, rather than a Str.

Another thing TimToady did was make the “…” operator (for smart sequences) behave more correctly with regards to strings.

Thanks to lizmat, the say and note methods and subs now properly respect the “nl” setting on the IO object used.

Another thing I ought to mention is the stellar ticket and testing work that bartolin has been doing for what feels like many months.

Documentation

There’s now a web site examples.perl6.org with lots of example code for you to peruse!

On the doc.perl6.org site, there’s now language documentation about control flow statements as well as exceptions.

Parting words

One other thing worth mentioning is that you now get a perl6-gdb-m and perl6-valgrind-m script installed with your rakudo for quickly running your stuff under gdb or valgrind.

And with that I’m already finished with the weekly. This time it’s done before midnight local time, too!

I wish you a pleasant week; until next time!

2015.14: Happy easter, enjoy your unsigned types!

Please excuse the delay; turns out the Easter weekend was a bit more busy than I expected it to be, and getting the post out on tuesday wasn’t so successful, either. I hope yours was just as nice as mine, in any case :)

I’ll report the changes from the perspective of monday (monyay!) anyway, since otherwise the next weekly might end up looking quite meager ;)

There’s already a few nice things I could have reported post-monday, so stay tuned for next week!

NativeCall improvements and unsignedness all around

NativeCall used to give errors when it found things like “uint” or “uint8″ in its argument lists. That has been fixed on both MoarVM and the JVM! For some reason, int64 was also not supported in NativeCall, but it now is. Also, why not use “longlong” or “ulonglong” every now and then? ;)

In other NativeCall news, parameters marked “is rw” (for int and num types) now work! You can have a look at the test file and the corresponding .c file to get a feel for how this works.

We’ll catch you when you fall …

There’s a few more cases where forgetting you’re writing Perl 6 code is gracefully caught by the compiler. For example, starting a file with a “package” declaration, or writing “elseif” or “else if” or “elif”. Now there’s “commatraps” where perl5 allowed writing something like “map { $_ + 5 } 1, 2, 3″ where there’s no need to put a comma after the block. Rakudo will remind you of the “Missing comma after block argument to map” if you write code like that.

In addition, something that comes up somewhat often is confusing the Whatever Star with a simple $_. Writing code like “map { * + 5 }, 1, 2, 3″ results in a list of three WhateverCode objects, because “* + 5″ already means something like “{ $_ + 5 }”, thus turning the whole thing into basically “map { { $_ + 5 } }, 1, 2, 3″. The current error message for that reads “Malformed double closure; WhateverCode is already a closure without curlies, so either remove the curlies or use valid parameter syntax instead of *”. A bit wordy, but I think it helps you towards fixing your code.

Another new error-related feature is that if you get a “Missing or wrong version of dependency of …” error, it’ll now also tell you what exact module had the dependency on the missing thing. This’ll make it much easier to track what you need to recompile/reinstall in order to get things back into working order.

A listy change

TimToady has started “undistinguishing” Parcel and List, so that the two types can become one for the Great List Refactor.

In case you don’t remember, the Great List Refactor is going to make lists (and lists of lists, and itemized lists) and flattening behavior of subs, methods and operators more uniform and easy to remember, but also improve the performance of things like list iteration a whole lot – currently there’s a whole lot of administrative overhead involved in for loops and such …

A performance hit

Sadly, merging the S11 changes that introduce a more database-like way of installing and storing packages made startup quite a bit slower. That’s because we now have a JSON “database” that we have to parse during startup. Sadly, the performance of JSON::Tiny (which is pretty much exactly what we use in CompUnitRepo::Installation) is not terribly good. In the future, the “database” will most likely be precompiled and loading time should be drastically better.

Miscellaneous stuff

There’s of course a few performance improvements again, but nothing mind-blowing this time.

On the ecosystem side of things, there’s a few new modules:

  • overwatch monitors your program’s dependencies and restarts the process if they have changed
  • Pastebin::Shadowcat posts to and retrieves pastes from the pastebin hosted on scsys.co.uk
  • App::Kains is a tool to constrain programs to a namespace-based container on linux
  • EventEmitter adds a coating of sugar on top of Supply to give a subscription interface to any object that allows arbitrary named events to be used

I think that’s most of the things that have happened over the past week; Since I’m already quite late and also running out of energy a little bit, I’ll just end it here. Feel free to leave anything I’ve missed in the comments section below :)

2015.13: The first weekly of 2015 that won’t be mistaken for a monthly!

Hi! This week marks the first week this year where the week number (as per ISO something or other) that’s bigger than 12. A few weeks ago, I got up in the middle of the night and read on the IRC that someone was surprised about the number of the weekly i had just released before going to bed. I was so surprised (and still tired) that i immediately “fixed” the number to what the current month was. Silly me!

Anyway, here’s what’s new in the land of the perls and moars:

REPL improvements

hoelzro has been working on removing linenoise – the readline replacement we’ve been using in MoarVM that’s been a bit troublesome with regards to windows support and multi-byte characters – from MoarVM core and re-introducing it as a Perl 6 module, this time even with tab completion. In addition to that, psch has been poking at making the REPL understand when a statement has only been partially entered and when it should ask for more text to finish the statement. This will make using the REPL so much more comfortable once it’s landed!

Properness improvements

TimToady has spent some time deep in the grammar and a few operators/metaoperators here and there to make parsing more robust and correct, give better errors (or get us back to previously good error messages in regressed cases) and also made the reduction meta-operator, when applied to a list infix metaoperator like Z or X able to work properly with lists of lists, even when they are specified indirectly.

jnthn just fixed a bunch of problems with \( … ) syntax for Captures. It now accepts ; and differentiates properly between Pair objects and named parameters and also accepts flattening into the Capture with the | prefix operator.

jnthn also advanced the state of native arrays and their serialization (which mostly causes pre-compiled modules to work better) on both MoarVM and JVM. Serialization of parameterized types is also partially in now, but interning those types isn’t implemented yet.

Staying on the MoarVM side of things, rudi_s appeared on the IRC channel and dropped off a bunch of patches to make our code nicer: unused functions and variables were thrown out, functions that don’t return or where the result ends up unused were properly marked, format strings were fixed. I don’t know what rudi’s involvement will end up being, but I hope when the codebase feels more familiar to them, they’ll make some deeper changes :)

Performance improvements

While working on making “use nqp” a mandatory declaration for using nqp:: ops in modules, moritz found out that “nqp::time_n” is a very popular op across our ecosystem. Why is that, you ask? Because “now”, which gives you a posix timestamp with fractions-of-seconds accuracy, is terribly slow. Actually, I should say “was” rather than “is”. jnthn found out that a big chunk of the time spent in “now” comes from the coercer that turns real values (Num) into rational values (Rat). That thing is now about 6.5x faster than it used to be.

jnthn also had a close look at how spesh behaves during startup and tuned the specializer and on-stack-replacement thresholds in MoarVM to reduce overhead during startup. This impacts both memory usage and time usage.

The Ecosystem

In the ecosystem there’s now the beginnings of a Git::PurePerl port from p5 to p6, which can already clone and checkout repositories, albeit very slowly. During the week two other modules showed up, namely Linux::Fuser and Browser::Open.

Another interesting thing to point out is that thanks to a bit of work by retupmoca GTK::Simple will now download DLL files from gtk-dlls.p6c.org if they can’t be found on a windows system. This makes deploying GTK-using Perl 6 programs to windows systems much simpler. I already asked retupmoca to apply the same treatment to a few more modules.

I personally spent a bit of time playing around with HTTP::Server::Async and i’ve learned a thing or two about javascript, HTML5 technologies (in particular EventSource, which is kinda cool) and how to glue it together with Perl 6. A few things have come up that ought to be changed in HTTP::Server::Async, but tony-o (HTTP::Server::Async’s author) is already on the case.

“eleven” lands for panda!

FROGGS has merged a panda branch called “eleven”, which makes use of the CompUnitRepo functionality of rakudo. The effect is that installed distributions will not be stored as files on disk, but go into a database like thing. This is done to be able to install (and locate) several same-named versions of a distributions, or distributions made by different authors. These can now coexist, though as a consequence every distribution must have a “provides” section [^1] in its META.info.
Such a “provides” section (or an entire META.info) can be generated by running “panda gen-meta”. Modules without a proper “provides” have a panda icon with a strike-through “S11″ flag next to them on the modules list, so if you want to dip your toe into contributing to Perl 6, here’s a very simple way to do so… :)
The CompUnitRepo database is not just there to allow looking up namespaces by distribution version or author, it also copes with the issue that arises when two distributions try to provide a binary/script with the same name. The CompUnitRepo will install wrapper scripts instead, which take –auth, –name and –ver options to select a specific binary or script to execute. If the search result is ambiguous a list of candidates is displayed to allow to refine the search criteria.
Next on FROGGS’s TODO is to implement %?RESOURCE [^2] and also to look into any fallout that happened because of his latest work.

[^1] http://design.perl6.org/S22.html#provides
[^2] http://design.perl6.org/S22.html#resource

And just like that, another week has already come and gone. This week, I’ll actually have a look at finding some interesting snippets for you to post on a different day. I’m thinking Thursday would be a good day to try. Thanks for the warm words, everyone :)

See you on Thursday, or at least on Monday :)