2015.20&21: Blogs, Grant, Release and Progress

The past weeks we’ve seen quite a few nice Perl 6 related blog posts. Just in case you haven’t seen them, I would like to repeat them here for you:

The Annotated Berlin Consensus

At the 2015 QAH in Berlin, a consensus was reached on a number of Perl toolchain issues, with a particular focus on governance and recommended standards of care for CPAN authors. The official version in Markdown can be found on Github. David Golden also blogged an annotated version of it. Although one could argue this is not strictly Perl 6 related, all of the issues mentioned in the consensus, will come to the Perl 6 ecosystem sooner or later.

Advancing the MoarVM JIT

This grant proposal by Bart Wiegmans (brrt) has been accepted. This is great news, can’t wait for bare Rakudo startup time to drop below 0.05 seconds!

Bare startup time below 0.1 second

In the past week, the bare startup of Rakudo on MoarVM finally dropped below 100 milliseconds on most developers’ machines. This is now only a factor of 20 slower than Perl 5. However, I’m told that starting Perl 5 with Moose (which from a feature point of view, gets it closer to Perl 6), is now slower than starting Rakudo!

Dresden

We also had the 2015.05 compiler release of Rakudo, named Dresden (after the recent German Perl Workshop there). The list of changes is quite extensive, coming in at 370 commits since the last release.

Changes since the Dresden release

  • use trace: lexical pragma to trace program execution on STDERR
  • Empty IO::Path / “”.IO no longer allowed, it used to be equivalent to “.”

GLR underway, watch your modules!

One can say that the GLR (Great List Refactor) is now really underway. Several modules in the ecosystem are now known to fail because of GLR induced changes. If you have a module in the Perl 6 ecosystem, it would be nice if you could check it for GLR induced bitrot using the latest Rakudo!

Shutting down

This Perl 6 Weekly came in a little late. Hopefully, future Perl 6 Weeklies will come in a more timely manner. Until then, keep up the good work!

2015.19: Events & Conferences

During the last week, there were multiple conferences where perl6ers could be found:

  • Larry Wall gave a talk at Collision in downtown Las Vegas and a Q&A session afterwards using twitter and an app called pundit. I have no idea if there’ll be recording of the talk, but supposedly, you’d be able to get at the Q&A stuff somehow somewhere …
  • At the German Perl Workshop froggs talked about native library bindings, using libxml2 as an example.
  • Also at the GPW, lichtkind gave a talk about functional programming in perl6, but the slides are in german.
  • Another talk from the GPW is rurban’s overview over VMs related to perl. The talk and the slides are in english.
  • At the OSDC.no there were many perl 6 related events, which can all be found on the “list of talks tagged Perl 6“. Sadly, no links to slides can be found there, and I don’t yet know about recordings.

But there was also some work being done – there were post- and/or pre-conference hackathons, too.

  • froggs threw pretty much all files that were in our roast repository into rakudo’s spectest.data, which means that a whole bunch of test files weren’t actually being run until now. Those are all fudged properly now, though, so that we can further decide to throw out tests that haven’t stood the test of time, for example. [correction: actually a bunch of those are still not being run, but at least got triaged]
  • TimToady added line number annotations to warnings more often, and also hunted down a bunch of “leaking” Failures, which means a Failure gets created at some point, but never sunk or inspected. (remember, Failures are basically unthrown exceptions)
  • lizmat has improved @*INC and the “use cur” pragma (which is going to replace “use lib” when it works as wanted – of course it’ll also get the “use lib” name at that point) a whole lot.
  • TimToady is moving forward to make Nil no longer disappear when doing list flattening or assignment. Right now, there’s an Empty value that’ll do the “disappears in flattening” and Nil no longer disappears in lists. If you yourself want this behavior, please use () instead, as Empty is most probably going to disappear once the “switch-a-roo” is done :)
  • thanks to raydiak “undeclared variable” errors now show a more correct error position and don’t give you the list of what kinds of rules/tokens the parser was expecting at the point of failure
  • lizmat restored the performance of IO::Handle.words and .lines when providing the :eager parameter, that you can use to not pay the “lazyness cost”. Of course, this is what we want the GLR to “automatically” provide in most cases: not paying the lazyness cost if you’re actually eagerly using something that can be lazy if it wants to (or if you want to).
  • jonathanstowe improved the “earliest” block which now handles having both “more $foo” and “more *” blocks in it.
  • brrt gave me the go-ahead to merge the “jit_devirtualize_reprops” branch, which turns indirect calls to a bunch of operators on objects into direct calls (if the type is known at jit-time). This is also something we’d have wanted for brrt’s sponsored work on improving the MoarVM jit further.
  • froggs is quite far on the way towards getting “ignoremark” support into rakudo, which lets a regex (or parts thereof) match regardless of any combining characters applied to the target string. [correction: in fact, froggs is so far along the way that ignoremark is already in rakudo!]
  • jonathanstowe added some more concurrency related docs to our documentation repository (and website).
  • moritz described how to get default values at object creation time into your classes in the classtut

That’s all I have for now. There wasn’t quite as much progress with so many perl6ers traveling and attending or giving talks and workshops.

And at 666 words (not including this paragraph), I feel this is a good place to stop writing. Have a nice week, everybody!

2015.18: Startup time rectified, hashes unordified, startup speedupified, puns deduplified

This week pleases me greatly, because there’s performance improvements to report!

Startup time & performance

lizmat worked hard to make initialization of the module installation database in rakudo lazy, so that any script that doesn’t “use” any module won’t take the performance hit. This (together with a bunch of other improvements) gets startup time (as measured by compiling and running an empty program) back to 0.17s on my machine (previously it was 0.31s) and the memory usage went from about 85 megs to 69 megs (these numbers come from the time command outputting the maximum resident set size). On lizmat’s machine, that reduced the time it took to do a full spectest run from 1620 to 1435 CPU seconds.

On the other side of the module installation database coin, froggs and jnthn just had a little breakthrough that will finally allow the json database to be replaced with a MoarVM serialization blob. Needless to say, this will be a whole bunch faster to load and store when we actually do need to load the database.

On top of these big improvements there’s of course been small improvements again. After profiling found out that a lot of time is spent decoding utf8 during startup, jnthn switched a bunch of things that are known to always be valid latin1 to decode that instead. This reduces the instructions run during startup by about 10%.

Another nice change is that .moarvm files used to contain not only a string heap section but also a chunk of code that used to create a list of all these strings. Now the string heap is used for both purposes, shaving a good 60 kilobytes off the CORE.setting.moarvm (still about 9 megabytes big) and reducing the startup time by about 2%. And then there’s a few sundry changes that all shave off a few percent here and there.

Adding to this heap of performance improvements all around, hashes on MoarVM no longer preserve insertion order. This feature – which users were discouraged from relying on anyway – was implemented in uthash by a doubly-linked list. jnthn removed that doubly linked list and made hashes save a noticable bit of memory: 16 bytes per hash entry and another 8 bytes per hash.

In the future, uthash will be more heavily modified so that strings representing the same text in different storage formats (8 byte per character vs full NFG) will result in the same hash. Then we’ll finally be able to store simpler strings more compactly. I’ve been looking forward to this for a long time, and now it seems to be within reach :)

Something I forgot to mention last week is that a whole bunch of methods on Str are now much faster because they have gotten their own implementation rather than relying on Cool to implement them (and do a needless coercion first in every case). This change was inspired by making .chars much faster in the very same way.

Bug Fixes

There used to be a bug where you’d get maddening errors like this:

Calling rl_get_screen_size(TypedPointer[Int], TypedPointer[Int]) will never work with declared signature (TypedPointer[Int], TypedPointer[Int])

This came from NativeCall using the “make_pun” meta-object method that erroneously created a fresh, distinct class each time TypedPointer[Something] was used somewhere. That method call has now been replaced with just “pun”, which goes through a cache first.

ptc found and fixed a few small memory leaks in MoarVM.

Today jnthn spent time on known multithreading-related instabilities and oddnesses. A big part of that was fixing a race in the Fixed Size Allocator which is used for frames, on-stack-replacement, stuff like that. There was a CAS-based lock that suffered from an ABA problem, which basically means one thread interpreted “two changes” (there and back) as “no change” (“go ahead”) and the list of free spots in the allocation pool could get corrupted, leading to memory corruption in objects when two threads use the same spot for different types of objects. The multi dispatch cache also just got a lock added to it, too. There is still more waiting to be found and fixed, though.

I personally got a tiny bit of work in, as well: If you ever encountered the error “Can’t adverb that!”, you’ll now find that it’ll also give a little hint as to what exactly the parser thought you were trying to adverb (because the precedence level of adverbs takes a little getting used to, IMO).

There used to be a problem related to statement modifiers and the given and for keywords where the value of $_ would get messed up, and block-less “gather” sometimes didn’t work where just adding curly braces would fix the problem. Thanks to jnthn, that’s fixed now as well.

Implemented features

Thanks to lizmat variables can now take “will” blocks, which correspond to phasers. Those already implemented are enter, leave, keep, undo, first, next, last, pre and end. Also, unimplemented “will” types now throw an error instead of failing silently. In addition to that, “lazy attributes” are now supported in rakudo. These let you write code like this:

class Example {
    has $.attr will lazy {
        # do some expensive set-up here
        "hello" # value returned from this block will be used
    }
}

When the attr is first accessed, it will run the code to set up the value and after that it will behave like any regular attribute.

After making List.roll faster last week, lizmat also made List.roll(*) (and List.roll(Inf)) return a list that knows it’s infinite.

If you set the environment variable RAKUDO_VERBOSE_STACKFRAME to a number, that number of lines of code around a line indicated in a stack trace will be shown. Another change to stack traces is that instead of a few select routines from the core setting, all routines in the core setting will now be hidden from backtraces by default – you can get full backtraces with the –ll-exception flag as before.

Even though this isn’t strictly a feature, there’s also been more changes regarding the GLR – mostly how flattening and nonflattening happens in relation to “for” and “map” in both their sub and method form. Lists and Parcel have been moving closer and closer and soon the Parcel class will be thrown out.

Ecosystem goodness and documentation

Even though recent changes to the module installation system and making hashes unordered brought a few bugs and problems out of hiding, there’s more nice stuff to be found in the ecosystem now!

  • Module::Minter gets your own module started with the basic files every such project needs (by dnmfarrell)
  • Template::Protone lets you write templates that compile down to fast perl6 subs and let you easily embed perl6 code (by tony-o)
  • ANTLR4::Grammar lets you parse an ANTLR4 grammar and use it to parse stuff. (Though it’s not been put into the ecosystem yet, I decided to highlight it here none the less.) (by DrForr)

ptc has pushed a whole bunch of commits to the perl6 examples site and ptc, stmuk, jonathanstowe and moritz have been adding and modifying more and more stuff on the perl6 documentation site.

labster wrote a blog post “Three Tales of Second System Syndrome” that he put up on his site, hoping to get a bit of feedback before spreading it further. However, it reached Hacker News only shortly after he went to bed and there’s now 121 comments in the corresponding thread.

That’s it for today!

Very strange; even though it feels like I’ve written the longest post in a few weeks, I also have a sneaking suspicion that I’ve forgotten to mention multiple things!

In any case, it’s time for me to finally finish up and maybe get back to annotating MoarVM’s code with a few hints for Valgrind and its tools. Have a great week!

2015.16 + 17: A release, NFG and more GLR work

Hello everybody! I’m back after a postless week. Since there was a compiler release in the earlier of the two weeks, I’ll highlight some of the things this compiler release brings and advances and changes made since then.

Sadly, there’s currently a major startup performance regression that has to do with the way we install modules and their files into a database-like format. This will most likely mean there’ll not be a Rakudo Star release this month.

Pragmatism

Pragmas (like “use soft” or “no strict” or “use fatal”) now properly work with lexical scoping.

Along with proper scoping of pragmas, “use fatal” now has a proper implementation. What that implies is for example “try 1/0″ immediately blowing up rather than appearing to succeed and returning a Failure object.

The third thing I can mention is that -Mstrict now works to make programs passed on the commandline with -e strict. In fact, -M works with all pragmas now.

Performance improvements

One of the first big improvements came from nwc10 going over MoarVM’s serialization format and kicking out a whole bunch of unnecessary 0 bytes all over the place and also making the deserialization step take fewer CPU cycles in general. This leads to noticably smaller .moarvm files, for example, but also slices off a few cpu cycles during startup. We’re still quite a bit away from getting down to 0.01s startup, compilation and execution for an empty program, but it seems we’re getting closer and closer each month.

By not trying to hold the complete output in RAM, MoarVM’s profiler is now able to spit out json blobs so gigantic that not even the profiler data analysis program written in C++ to handle more data will open it. Before that change (many thanks to flussence for this, btw!) MoarVM would very quickly balloon up in its memory usage and get OOM killed.

lizmat built a eager variant of List.pick(*) that’ll give you a shuffled version of a list faster than the lazy variant. However, List.pick and List.pick(*) themselves got 3% and 7% faster. Something else that’s gotten faster thanks to lizmat is the clone method for the cases where no changes are made to the resulting object via named parameters.

Documentation

On the documentation side, doc.perl6.org received a 857 inserted lines over the two week period covered. Thanks to moritz, ptc, stmuk, skids and jonathanstowe for their contributions! The files that got changed the most were concurrency, control and objects in the language section. If you miss something in the doc repo or the website, feel free to ask on the IRC channel, open an issue or send a pull request for the WANTED file. Of course if you like, you can also send pull requests for changes or additions to the docs directly.

moritz published a blog post about writing docs on his blog that I found interesting.

And then there’s examples.perl6.org which has seen many, many commits from ptc and stmuk. Kudos!

NFG, GLR, other cool bits

I’d write about the NFG things jnthn has implemented, but he already blogged about it on his personal blog and explained a lot, so I’ll just send you over there to read. If you’re running rakudo from the git repository, you’ll have the Str class backed by NFG. If you’re running the latest compiler release, you’ll be able to use .NFG on a Uni object, but NFG isn’t the default yet.

On the GLR side of things, TimToady has been working more on handling things that flatten or stay itemized. It’s hard to turn the changes and discussion into a succint summary, though. One thing that’s gotten noticably more consistent is the sequence operator (“…”). The latest discussion was on how the two forms of looping over an array should be called: One that treats a list as a collection of items and one that causes a list to flatten its inner lists and iterate over the result.

The rotor method of List and Supply now take a list of pairs as their arguments in the format Take-How-Many => Move-How-Many-More. For now, if you supply two integers, you’ll get the old format of Take-How-Many, Step-Back-How-Many [correction: the backwards compatibility mentioned in this sentence only works in the released version]. This makes it more sensible (the previous default value for Step-Back-How-Many was 1, which is sort of unexpected).

psch and hoelzro have both invested time in the REPL. psch made the tr/// operator aware of adverbs, and of course psch has spent more time on the JavaInterop side of rakudo.

You can now assign a finite list to an infinite slice of an array or assign an infinite list into a finite slice of a list, like this:

my @the-list;
@the-list[15..*]  = 1, 0, 1, 0, 1, 0;
@the-list[0..10] = 0, 2 ... *;
say @the-list;
#-> 0 2 4 6 8 10 12 14 16 18 20 (Any) (Any) (Any) (Any) 1 0 1 0 1 0

As for froggs, have a look at this bit of code he made work just today (well, yesterday):

> say (0..0xFFF0).chrs.join('') ~~ m:g/<:NumericValue({+$_ > 0})>+/
「123456789」 「²³」 「¹」 「¼½¾」 「١٢٣٤٥٦٧٨٩」 「۱۲۳۴۵۶۷۸۹」 「߁߂߃߄߅߆߇߈߉」 「१२३४५६७८९」 「১২৩৪৫৬৭৮৯」 「৴৵৶৷৸৹」 「੧੨੩੪੫੬੭੮੯」 「૧૨૩૪૫૬૭૮૯」 「୧୨୩୪୫୬୭୮୯」 「୲୳୴୵୶୷」 「௧௨௩௪௫௬௭௮௯௰௱௲」 「౧౨౩౪౫౬౭౮౯」 「౹౺౻౼౽౾」 「೧೨೩೪೫೬೭೮೯」 「൧൨൩൪൫൬൭൮൯൰൱൲൳൴൵」 「෧෨෩෪෫෬෭෮෯」 「๑๒๓๔๕๖๗๘๙」 「໑໒໓໔໕໖໗໘໙」 「༡༢༣༤༥༦༧༨༩༪༫༬༭༮༯༰༱༲」 「၁၂၃၄၅၆၇၈၉」 「႑႒႓႔႕႖႗႘႙」 「፩፪፫፬፭፮፯፰፱፲፳፴፵፶፷፸፹፺፻፼」 「ᛮᛯᛰ」 「១២៣៤៥៦៧៨៩」 「៱៲៳៴៵៶៷៸៹」 「᠑᠒᠓᠔᠕᠖᠗᠘᠙」 「᥇᥈᥉᥊᥋᥌᥍᥎᥏」 「᧑᧒᧓᧔᧕᧖᧗᧘᧙᧚」 「᪁᪂᪃᪄᪅᪆᪇᪈᪉」 「᪑᪒᪓᪔᪕᪖᪗᪘᪙」 「᭑᭒᭓᭔᭕᭖᭗᭘᭙」 「᮱᮲᮳᮴᮵᮶᮷᮸᮹」 「᱁᱂᱃᱄᱅᱆᱇᱈᱉」 「᱑᱒᱓᱔᱕᱖᱗᱘᱙」 「⁴⁵⁶⁷⁸⁹」 「₁₂₃₄₅₆₇₈₉」 「⅐⅑⅒⅓⅔⅕⅖⅗⅘⅙⅚⅛⅜⅝⅞⅟ⅠⅡⅢⅣⅤⅥⅦⅧⅨⅩⅪⅫⅬⅭⅮⅯⅰⅱⅲⅳⅴⅵⅶⅷⅸⅹⅺⅻⅼⅽⅾⅿↀↁↂ」 「ↅↆↇↈ」 「①②③④⑤⑥⑦⑧⑨⑩⑪⑫⑬⑭⑮⑯⑰⑱⑲⑳⑴⑵⑶⑷⑸⑹⑺⑻⑼⑽⑾⑿⒀⒁⒂⒃⒄⒅⒆⒇⒈⒉⒊⒋⒌⒍⒎⒏⒐⒑⒒⒓⒔⒕⒖⒗⒘⒙⒚⒛」 「⓫⓬⓭⓮⓯⓰⓱⓲⓳⓴⓵⓶⓷⓸⓹⓺⓻⓼⓽⓾」 「❶❷❸❹❺❻❼❽❾❿➀➁➂➃➄➅➆➇➈➉➊➋➌➍➎➏➐➑➒➓」 「⳽」 「〡〢〣〤〥〦〧〨〩」 「〸〹〺」 「㆒㆓㆔㆕」 「㈠㈡㈢㈣㈤㈥㈦㈧㈨㈩」 「㉈㉉㉊㉋㉌㉍㉎㉏」 「㉑㉒㉓㉔㉕㉖㉗㉘㉙㉚㉛㉜㉝㉞㉟」 「㊀㊁㊂㊃㊄㊅㊆㊇㊈㊉」 「㊱㊲㊳㊴㊵㊶㊷㊸㊹㊺㊻㊼㊽㊾㊿」 「㐅」 「㒃」 「㠪」 「㭍」 「一」 「七」 「万」 「三」 「九」 「二」 「五」 「亖」 「亿什」 「仟」 「仨」 「伍」 「佰」 「億」 「兆」 「兩」 「八」 「六」 「十」 「千卄卅」 「卌」 「叁参參叄」 「四」 「壱」 「壹」 「幺」 「廾廿」 「弌弍弎」 「弐」 「拾」 「捌」 「柒」 「漆」 「玖」 「百」 「肆」 「萬」 「貮」 「貳」 「贰」 「阡」 「陆」 ...

As you can see, you can now match against a character’s unicode property value inside a regex. In this case we’re looking for unicode characters that have a numerical value that’s bigger than 0. Pretty cool, IMO!

On the ecosystem, there’s now a module that lets you write a grammar definition in BNF form and use it just like a Perl6 Grammar (that’s because it actually generates a Perl6 Grammar class for you). It’s made by tadzik and you can have a look at one of its test files here. There’s also a POD parser (for the perl5 variant of POD) by dnmfarell on github and the ecosystem.

The last thing before I finally go to bed will be the release itself. We have masak to thank for the release process itself, but I was very glad to see so many different people work on so many different pieces of the whole. I may only be making this up, but it feels like we’ve gotten more contributors :)

Well, at least the weekly is pretty big this time ’round, and I don’t think that’s because it covers twice as much time as usual, and there’s already more cool things to look forward to in the coming week :)

So you all have a great week now!

2015.15: We figured out the truth about “0”!

Jonathan Workington

jnthn just published a new blog post last night describing the fund that was recently set up for Perl 6 development (thank you, WenZPerl!) that’ll be used among other things for funding more jnthn time until the release coming in december. There’s also a lot of information on what jnthn has recently been doing in that blog post:

  • Unicode database in MoarVM updated to version 7.0
  • unsigned types in NativeCall (already mentioned in the previous p6weekly)
  • a whole lot of RT tickets fixed, like improved error messages for BEGIN-time things and constants, “where” constraints on attributes and variables
  • the beginning of the Uni class and its features, like different normalization forms (as specified by Unicode)

Performance improvements

Nicholas Clark has been churning out patches to reduce the number of cpu instructions “used up” during rakudo’s startup and also improved the efficiency of our serialization format.

TimToady improved the MATCH method in both rakudo and nqp that are responsible for creating the high-level object that gets created for a regexp match. That’s quite a hot method and initial, noisy measurements showed it shaves off about 5% of the time spent in the “stage parse” of rakudo’s core setting.

In addition to MATCH being faster now, rakudo (and nqp) used to flip ginormous amounts of string data whenever it encountered a <?after …>, but now [correction:] if the length of the pattern inside is known, it turns into a <?before …> with the pattern itself flipped.

I’ve built support for a whole bunch of reprops (push/pop, elems, access to attributes, keys or positions, …) into the JIT for the cases where we couldn’t generate a super simple piece of assembly code to handle something (like pushing into a list that may have to enlarge the list) but we still know what exact function we’ll be calling. Unfortunately my desktop is currently not fully assembled, otherwise i’d have some benchmark data for you.

labster gave Str its own “chars” method (as opposed to inheriting it from Cool), which makes .chars on a String work about 40% faster. Wow!

Semantic improvements

Something that may have surprised a lot of people (myself included) was that Perl 6 claims that the truth value of “0” ought to be False. That is no longer the case, and so the rule for the truth value of strings is now just “does it have any characters in it?”. I think that’s more sensible.

TimToady improved the parser’s handling of heredocs in constant expressions and BEGIN blocks. There’s a bit of cheating necessary in the parser to make all of this work, but masak wrote a post on his blog about this very topic that I recommend you read.

TimToady also implemented Complex literals inside < >, which means that <3+2i>.WHAT now properly results in a Complex, rather than a Str.

Another thing TimToady did was make the “…” operator (for smart sequences) behave more correctly with regards to strings.

Thanks to lizmat, the say and note methods and subs now properly respect the “nl” setting on the IO object used.

Another thing I ought to mention is the stellar ticket and testing work that bartolin has been doing for what feels like many months.

Documentation

There’s now a web site examples.perl6.org with lots of example code for you to peruse!

On the doc.perl6.org site, there’s now language documentation about control flow statements as well as exceptions.

Parting words

One other thing worth mentioning is that you now get a perl6-gdb-m and perl6-valgrind-m script installed with your rakudo for quickly running your stuff under gdb or valgrind.

And with that I’m already finished with the weekly. This time it’s done before midnight local time, too!

I wish you a pleasant week; until next time!

2015.14: Happy easter, enjoy your unsigned types!

Please excuse the delay; turns out the Easter weekend was a bit more busy than I expected it to be, and getting the post out on tuesday wasn’t so successful, either. I hope yours was just as nice as mine, in any case :)

I’ll report the changes from the perspective of monday (monyay!) anyway, since otherwise the next weekly might end up looking quite meager ;)

There’s already a few nice things I could have reported post-monday, so stay tuned for next week!

NativeCall improvements and unsignedness all around

NativeCall used to give errors when it found things like “uint” or “uint8″ in its argument lists. That has been fixed on both MoarVM and the JVM! For some reason, int64 was also not supported in NativeCall, but it now is. Also, why not use “longlong” or “ulonglong” every now and then? ;)

In other NativeCall news, parameters marked “is rw” (for int and num types) now work! You can have a look at the test file and the corresponding .c file to get a feel for how this works.

We’ll catch you when you fall …

There’s a few more cases where forgetting you’re writing Perl 6 code is gracefully caught by the compiler. For example, starting a file with a “package” declaration, or writing “elseif” or “else if” or “elif”. Now there’s “commatraps” where perl5 allowed writing something like “map { $_ + 5 } 1, 2, 3″ where there’s no need to put a comma after the block. Rakudo will remind you of the “Missing comma after block argument to map” if you write code like that.

In addition, something that comes up somewhat often is confusing the Whatever Star with a simple $_. Writing code like “map { * + 5 }, 1, 2, 3″ results in a list of three WhateverCode objects, because “* + 5″ already means something like “{ $_ + 5 }”, thus turning the whole thing into basically “map { { $_ + 5 } }, 1, 2, 3″. The current error message for that reads “Malformed double closure; WhateverCode is already a closure without curlies, so either remove the curlies or use valid parameter syntax instead of *”. A bit wordy, but I think it helps you towards fixing your code.

Another new error-related feature is that if you get a “Missing or wrong version of dependency of …” error, it’ll now also tell you what exact module had the dependency on the missing thing. This’ll make it much easier to track what you need to recompile/reinstall in order to get things back into working order.

A listy change

TimToady has started “undistinguishing” Parcel and List, so that the two types can become one for the Great List Refactor.

In case you don’t remember, the Great List Refactor is going to make lists (and lists of lists, and itemized lists) and flattening behavior of subs, methods and operators more uniform and easy to remember, but also improve the performance of things like list iteration a whole lot – currently there’s a whole lot of administrative overhead involved in for loops and such …

A performance hit

Sadly, merging the S11 changes that introduce a more database-like way of installing and storing packages made startup quite a bit slower. That’s because we now have a JSON “database” that we have to parse during startup. Sadly, the performance of JSON::Tiny (which is pretty much exactly what we use in CompUnitRepo::Installation) is not terribly good. In the future, the “database” will most likely be precompiled and loading time should be drastically better.

Miscellaneous stuff

There’s of course a few performance improvements again, but nothing mind-blowing this time.

On the ecosystem side of things, there’s a few new modules:

  • overwatch monitors your program’s dependencies and restarts the process if they have changed
  • Pastebin::Shadowcat posts to and retrieves pastes from the pastebin hosted on scsys.co.uk
  • App::Kains is a tool to constrain programs to a namespace-based container on linux
  • EventEmitter adds a coating of sugar on top of Supply to give a subscription interface to any object that allows arbitrary named events to be used

I think that’s most of the things that have happened over the past week; Since I’m already quite late and also running out of energy a little bit, I’ll just end it here. Feel free to leave anything I’ve missed in the comments section below :)

2015.13: The first weekly of 2015 that won’t be mistaken for a monthly!

Hi! This week marks the first week this year where the week number (as per ISO something or other) that’s bigger than 12. A few weeks ago, I got up in the middle of the night and read on the IRC that someone was surprised about the number of the weekly i had just released before going to bed. I was so surprised (and still tired) that i immediately “fixed” the number to what the current month was. Silly me!

Anyway, here’s what’s new in the land of the perls and moars:

REPL improvements

hoelzro has been working on removing linenoise – the readline replacement we’ve been using in MoarVM that’s been a bit troublesome with regards to windows support and multi-byte characters – from MoarVM core and re-introducing it as a Perl 6 module, this time even with tab completion. In addition to that, psch has been poking at making the REPL understand when a statement has only been partially entered and when it should ask for more text to finish the statement. This will make using the REPL so much more comfortable once it’s landed!

Properness improvements

TimToady has spent some time deep in the grammar and a few operators/metaoperators here and there to make parsing more robust and correct, give better errors (or get us back to previously good error messages in regressed cases) and also made the reduction meta-operator, when applied to a list infix metaoperator like Z or X able to work properly with lists of lists, even when they are specified indirectly.

jnthn just fixed a bunch of problems with \( … ) syntax for Captures. It now accepts ; and differentiates properly between Pair objects and named parameters and also accepts flattening into the Capture with the | prefix operator.

jnthn also advanced the state of native arrays and their serialization (which mostly causes pre-compiled modules to work better) on both MoarVM and JVM. Serialization of parameterized types is also partially in now, but interning those types isn’t implemented yet.

Staying on the MoarVM side of things, rudi_s appeared on the IRC channel and dropped off a bunch of patches to make our code nicer: unused functions and variables were thrown out, functions that don’t return or where the result ends up unused were properly marked, format strings were fixed. I don’t know what rudi’s involvement will end up being, but I hope when the codebase feels more familiar to them, they’ll make some deeper changes :)

Performance improvements

While working on making “use nqp” a mandatory declaration for using nqp:: ops in modules, moritz found out that “nqp::time_n” is a very popular op across our ecosystem. Why is that, you ask? Because “now”, which gives you a posix timestamp with fractions-of-seconds accuracy, is terribly slow. Actually, I should say “was” rather than “is”. jnthn found out that a big chunk of the time spent in “now” comes from the coercer that turns real values (Num) into rational values (Rat). That thing is now about 6.5x faster than it used to be.

jnthn also had a close look at how spesh behaves during startup and tuned the specializer and on-stack-replacement thresholds in MoarVM to reduce overhead during startup. This impacts both memory usage and time usage.

The Ecosystem

In the ecosystem there’s now the beginnings of a Git::PurePerl port from p5 to p6, which can already clone and checkout repositories, albeit very slowly. During the week two other modules showed up, namely Linux::Fuser and Browser::Open.

Another interesting thing to point out is that thanks to a bit of work by retupmoca GTK::Simple will now download DLL files from gtk-dlls.p6c.org if they can’t be found on a windows system. This makes deploying GTK-using Perl 6 programs to windows systems much simpler. I already asked retupmoca to apply the same treatment to a few more modules.

I personally spent a bit of time playing around with HTTP::Server::Async and i’ve learned a thing or two about javascript, HTML5 technologies (in particular EventSource, which is kinda cool) and how to glue it together with Perl 6. A few things have come up that ought to be changed in HTTP::Server::Async, but tony-o (HTTP::Server::Async’s author) is already on the case.

“eleven” lands for panda!

FROGGS has merged a panda branch called “eleven”, which makes use of the CompUnitRepo functionality of rakudo. The effect is that installed distributions will not be stored as files on disk, but go into a database like thing. This is done to be able to install (and locate) several same-named versions of a distributions, or distributions made by different authors. These can now coexist, though as a consequence every distribution must have a “provides” section [^1] in its META.info.
Such a “provides” section (or an entire META.info) can be generated by running “panda gen-meta”. Modules without a proper “provides” have a panda icon with a strike-through “S11″ flag next to them on the modules list, so if you want to dip your toe into contributing to Perl 6, here’s a very simple way to do so… :)
The CompUnitRepo database is not just there to allow looking up namespaces by distribution version or author, it also copes with the issue that arises when two distributions try to provide a binary/script with the same name. The CompUnitRepo will install wrapper scripts instead, which take –auth, –name and –ver options to select a specific binary or script to execute. If the search result is ambiguous a list of candidates is displayed to allow to refine the search criteria.
Next on FROGGS’s TODO is to implement %?RESOURCE [^2] and also to look into any fallout that happened because of his latest work.

[^1] http://design.perl6.org/S22.html#provides
[^2] http://design.perl6.org/S22.html#resource

And just like that, another week has already come and gone. This week, I’ll actually have a look at finding some interesting snippets for you to post on a different day. I’m thinking Thursday would be a good day to try. Thanks for the warm words, everyone :)

See you on Thursday, or at least on Monday :)

2015.12: A release. Also, lazy deserialization makes a comeback

On Friday, the 20th we got a Rakudo release, followed swiftly by a Star release on the 21st and a Rakudo Star .msi installer this monday. There’s also one Rakudo Star Docker Image based on Fedora 21 by mj41 and one Rakudo Star Docker Image based on Ubuntu 14.04 by hoelzro. On top of that, FROGGS wrote down how to create the Rakudo Star .msi packages. Big thanks to FROGGS for the Rakudo release and the MSI, moritz for the Rakudo Star release and everyone else involved in the Release process and all the tasks surrounding it!

Performance

A long time ago, MoarVM learned how to delay deserialization of a whole bunch of objects until they are referred to, but there was at least one bug that was caused by it and so it was disabled for a long while. That feature was re-enabled just after the release, so that we get quite some testing by the community to find out if it still causes issues and this time hopefully get rid of them, since the lazy deserialization gives us much better memory usage and speed when starting up a program that doesn’t use every class there is.

That’s not the only change we can look forward to, though. Jnthn is currently working on improving the impact of Frames on the garbage collector a whole lot, which (in my expectation) should improve memory usage as well as collection times.

In addition, Jnthn will fix my attempts at teaching NQP’s code-gen and Rakudo’s optimizer about references to local variables, which will recover the performance hit we took when our support for native variables and attributes has been vastly improved recently.

There was also a commit that prevented $*VM from being initialized on every startup that dramatically decreased startup time, but that caused a problem with installing modules, so it was reverted. Maybe we’ll end up getting it back this month with a proper fix for the underlying problem!

There’s also two things yours truly is working on: Making calls to a repr’s operations (attribute access, pushing/popping, positional and associative accesses, …) less overhead-ridden in the JIT compiler and merging facts about registers in spesh when Blocks in the Command Flow Graph come together. They aren’t ready for merging yet and also don’t seem to cause significant performance improvements, but I’ll keep looking for ways to make things faster …

Platform things

Rakudo used to follow parrot’s install directory layout, which put things in $PREFIX/languages. We changed that to $PREFIX/share instead; you may have to re-install modules and run Configure.pl again for all parts of your Rakudo installation if you’re staying on the bleeding edge via a git checkout. If things get very wonky with rakudobrew, consider nuking the subdirectories of ~/.rakudobrew …

This change was made to make rakudo and nqp much more friendly towards linux distributions.

Also, the docker image efforts by hoelzro and mj41 will be combined into a more official rakudo star docker repository.

What’s also very worth mentioning is Coke’s ongoing work to get our stuff into the official MacPorts repository. There’s only MoarVM so far, but NQP and Rakudo shall follow.

Language Features, Syntax, Bug fixes

usev6 fixed the interaction of “last” and the sequence operator, making sure that code like this

perl6 -e 'say 10, 9, 8, { $_ - 1 || last } ... *'
10 9 8 7 6 5 4 3 2 1 1

now properly drops the last entry in the list.

Raydiak (and Mouq) implemented the simplest case of an angle-bracketed Rat literal, like <1/5>. In the future, there’ll also be support for lists of Rat literals like <1/2 1/3 1/4>. I may have to bug them to also make Complex literals work in angle brackets :)

Psch fixed a problem in Rakudo-JVM’s Binder that’d cause coercive types to pass the original value rather than the coerced value to a function’s body. After that, he also implemented nqp::getuniname on Rakudo-JVM which powers Perl6’s uniname sub. A few improvements for the Eval Server have also landed.

Thanks to TimToady the “cannot call $sub_or_method, none of these signatures match:” error message also shows the capture that was passed and caused the “no match” exception. The robustness of Mu’s .perl method has also been improved and its output prettified a tiny bit.

Lastly, Mouq fixed the combination of Bag and hyper operators, such as “Bag.new(…) »-» 1″.

Another thing happening in a non-merged branch right now is more features and improvements for supporting CUnion meant for (but not limited to) NativeCall by FROGGS.

Documentation

moritz, ptc and others have been improving the official doc.perl6.org contents again. That’s Way Cool if you ask me! As always: feedback is very welcome on the IRC channel or the perl6-users mailing list, or even in the comments section of this blog.

Finishing up

I feel like I’ve missed a few things, but I’ve been procrastinating this post far too long now and bedtime is coming up soon :)

Therefor I wish you a nice week and have fun trying out the newest Rakudo Star release!

P.S.: I’m thinking about adding a second series of posts to this blog. Once a week I’d look through all the snippets evaluated with our evalbot on the IRC channel and pick out one or two and briefly describe what I like about them. What do you think about that idea?

2015.11 Native arrays arrive

Friday the Thirteenth turned out to be a very fruitful day, as jnthn++ could finally merge a lot of the work that he has been doing the past weeks in the background regarding native arrays. Native integer arrays come in the following flavours: int8, uint8 (one byte per element), int16, uint16 (two bytes per element), int32, uint32 (four bytes per element), and int64, uint64 (eight bytes per element). On top of that there are two generic native integer array types: int and uint, which are aliases for either the 32bit or 64bit version (depending on your hardware and/or build options). So what does that mean for performance in e.g. filling a 10M element array with an integer value:

my Int @A; my int $i;
while $i < 10000000 { @A[$i] = 1; $i = $i + 1 }   # 11.6 CPU seconds

Using an int array is more than 11x faster:

my int @a; my int $i;
while $i < 10000000 { @a[$i] = 1; $i = $i + 1 }   #  1.0 CPU seconds

If we compare this to Perl 5, we can see that Perl 5 is slower in this particular benchmark:

use integer;
my @a; my $i = 0;
while ($i < 10000000) { $a[$i] = 1; $i = $i + 1 } #  1.5 CPU seconds

However, Perl 5 optimizes the for idiom much better than using while. So if we use the for idiom, we get:

use integer;
my @a;
for (0 .. 9999999) { $a[$_] = 1 }             # 1.0 CPU seconds

So on par with Perl 5. Still, it is very exciting to see Perl 6 execution speeds approaching Perl 5’s!

Apart from native integer arrays, we also now have native numeric (real) arrays in 2 flavours: num32 (4 bytes) and num64 (8 bytes), and a generic num type that aliases to either.

The native arrays implementation is still a bit rough on the edges, specifically with regards to Listy functions such as .pick, .roll etc. The reason why these do not work yet, is clear: there is still some discussion as to how to resolve the issue, as it will involve some internal code shuffling.

Why is it so hard?

moritz++ wrote a very interesting blog post about why it is so hard to write a compiler for Perl 6. There were quite some comments on Reddit.

TheDamian in Amsterdam

Damian Conway gave his Everyday Perl 6 presentation in Amsterdam last week (like he did in Oslo the week before that). Some photo impressions from the lovely ambiance of the Tuschinski Theater. To quote TheDamian:

Perl 6 is a computer scientist’s dream come true. Its many advanced features (junctions, multiple dispatch, generics, delegation, grammars, lazy evaluation, infinite lists, hyperoperators, first class functions, coroutines, partial function application, macros, metadata, etc., etc.) offer the elite überhacker entirely new levels of awesome cosmic power.

Unfortunately, for the majority of us, those incredible new features are mostly just mysterious, scary, and off-putting. But all of those new features are also entirely optional. You don’t have to start out in Perl 6 writing autoparallelized infinite superpositions of multimorphic higher-order functions. You don’t even have to statically type your variables or encapsulate your objects.

Using attributes in regexes

jnthn++ added a compile time error when using class attributes inside a regex. From the commit message:

Regexes are methods on Cursor (or some subclass of it), meaning that
an attribute access inside of them is relative to that. Rather than
failing silently or exploding noisily at runtime, catch the problem at
compile time. The typed exception’s message explains the issue and
offers a suggestion.

Development Allsorts

  • FROGGS++ fixed a long standing bug on the JVM backend involving our scoped subs in combination with precompilation. This now allows e.g. the URI module to be used on the JVM backend.
  • hoelzro++ found a problem when multiple readers read from the same Channel. lizmat++ fixed this (at least for now) and hoelzro added a test for it.
  • lizmat kebab-cased some functionality (and added deprecations for the old names) in an effort to provide more naming consistency: is hidden-from-backtrace, is hidden-from-USAGE, use MONKEY-TYPING, RAKUDO-NO-DEPRECATIONS, Supply.on-demand, Supply.schedule-on.
  • If you’re running spectests regularly, you will see quite an increase in number of tests (36600+ tests now): these are mostly new native array tests.

Winding Down

timotimo++ was not feeling well enough yet to do the Perl 6 Weekly, so yours truly did it this time. timotimo, get and feel better soon!

Please let me know of any errors, comments, suggestions, like always!

2015.10 FOSDEM Videos and Role Conflicts

Unfortunately, timotimo++ wasn’t feeling well enough to do the Perl 6 weekly this week. So yours truly will be stepping in again. Best wishes to timotimo: hoping he will feel better again soon!

FOSDEM Videos

The FOSDEM videos of the Perl DevRoom have appeared online. I’d like to highlight the Perl 6 ones in this context:

Role conflict resolution, wraps and claim

skids++ has written a drive-by summary/followup thoughts on “the role debate”, after colomon++ had a blog post about it. Food for thought, indeed. Comments welcome, of course!

TimToady on a roll

While fixing many grammar related buglets, TimToady++ also made it illegal to create operators with spaces in them. A sane choice, if you ask me. He also specced and implemented three more pseudo-packages: LEXICAL (Contextual symbols in my or any outer’s lexical scope), OUTERS (Symbols in any outer lexical scope) and CALLERS (Contextual symbols in any caller’s lexical scope). The use of $/ as Perl 5’s input line separator, is now better detected and frowned upon with a clearer error message.

Array/Hash Internals Shakeup

It was discussed many times over the past weeks. This Sunday, lizmat bit the bullet by renaming all of the Rakudo specific array / hash access internal methods, and providing deprecation logic for the old method names. What does that mean? As a user, it shouldn’t matter much. As a module and/or internals developer, you may need to change some calls to these internal methods. Please check the ChangeLog for more information.

Nativecall C++ Integration

FROGGS++ has been steadily working on the integration of C++ with NativeCall in a branch. It is getting close to a merge, so hopefully we’ll be able to chalk up “C++ integration” on the list of NativeCall features next week.

Development Allsorts

The following features / bug fixes were implemented in the past week:

  • PairMap (Hash with fixed set of keys on creation, but variable values)
  • .pairup() (Create a list of pairs from a List of scalars, pairs, lists and hashes)
  • UInt (sub)type and coercer for unsigned integer.
  • is DEPRECATED works on subs/methods even if they don’t have phasers.
  • MoarVM gc crash that was blocking work in Inline::Perl5 has been fixed.
  • Most, if not all Expected Array[Str] but got Array[Str] errors fixed.
  • my Role $a .= new does not error out anymore.

And of course, many, many other bug fixes and optimizations, both on the MoarVM as well as specifically on the JVM backend (Mouq++, avuserow++, nwc10++, donaldh++ on top of the usual suspects).

Winding down

Please let me know of any errors, comments, suggestions, like always!