{"id":22,"date":"2021-04-15T06:00:00","date_gmt":"2021-04-15T06:00:00","guid":{"rendered":"https:\/\/school.moongames.eu\/?p=22"},"modified":"2022-07-14T13:55:26","modified_gmt":"2022-07-14T13:55:26","slug":"julia-hogwild","status":"publish","type":"post","link":"https:\/\/school.moongames.eu\/index.php\/2021\/04\/15\/julia-hogwild\/","title":{"rendered":"The Hogwild! idea"},"content":{"rendered":"\n

the Diploma thesis learning \u2013 April 15, 2021<\/p>\n\n\n\n

I have been working on a project based on the paper “HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent<\/em>“. <\/p>\n\n\n\n

https:\/\/papers.nips.cc\/paper\/2011\/file\/218a0aefd1d1a4be65601cc6ddc1520e-Paper.pdf<\/a><\/p>\n\n\n\n

First iteration was reading the paper and preparing the logical model how Stochastic Gradient Descent, a.k.a. SGD, works and how it would be possible to paralelize.<\/p>\n\n\n\n

Creating the Julia project<\/p>\n\n\n\n

@everywhere function run_SSP_worker(batch, server, channel, model, ps, loss)\n    id = myid()\n    println(\"$(id) has started!\")\n    while fetch(channel) != nothing\n        println(\"$(myid()) is bamboozled!\")\n        xs = take!(channel)\n        println(\"$(myid()) has\\n $(ps)\\n $(xs)\")\n        data = batch()\n        # WHY AM I NOT ALLOWED TO REMOVE MODEL HERE??\n        gs = calculate_gradient(loss, model, ps, xs, data)\n        information = Information(id, gs)\n        println(\"$(data)\\n # $(xs)\\n # $(gs)\")\n        @sync put!(server, information)\n    end\nend<\/code><\/pre>\n\n\n\n
Math code is \"E=mc^2\"<\/code><\/p>\n\n\n\n

At first, we sample \"f(x)\" in the \"N\" (\"N\" is odd) equidistant points around \"x^*\":
   <\/span>   <\/span>\"\[f_k = f(x_k),\: x_k = x^*+kh,\: k=-\frac{N-1}{2},\dots,\frac{N-1}{2}\]\"<\/p>
where \"h\" is some step.
Then we interpolate points \"{(x_k,f_k)}\" by polynomial
<\/a>
 (1) <\/span>   <\/span>\"\begin{equation*} P_{N-1}(x)=\sum_{j=0}^{N-1}{a_jx^j}\end{equation*}\"<\/p>
Its coefficients \"{a_j}\" are found as a solution of system of linear equations:
<\/a>
 (2) <\/span>   <\/span>\"\begin{equation*} \left{ P_{N-1}(x_k) = f_k\right},\quad k=-\frac{N-1}{2},\dots,\frac{N-1}{2}\end{equation*}\"<\/p>
Here are references to existing equations: (1<\/a>), (2<\/a>).
Here is reference to non-existing equation (??).
<\/p>\n","protected":false},"excerpt":{"rendered":"
the Diploma thesis learning \u2013 April 15, 2021 I have been working on a project based on the paper “HOGWILD!:<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[16,13],"class_list":["post-22","post","type-post","status-publish","format-standard","hentry","category-implementation","tag-julia","tag-ml","comments-off"],"_links":{"self":[{"href":"https:\/\/school.moongames.eu\/index.php\/wp-json\/wp\/v2\/posts\/22","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/school.moongames.eu\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/school.moongames.eu\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/school.moongames.eu\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/school.moongames.eu\/index.php\/wp-json\/wp\/v2\/comments?post=22"}],"version-history":[{"count":31,"href":"https:\/\/school.moongames.eu\/index.php\/wp-json\/wp\/v2\/posts\/22\/revisions"}],"predecessor-version":[{"id":168,"href":"https:\/\/school.moongames.eu\/index.php\/wp-json\/wp\/v2\/posts\/22\/revisions\/168"}],"wp:attachment":[{"href":"https:\/\/school.moongames.eu\/index.php\/wp-json\/wp\/v2\/media?parent=22"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/school.moongames.eu\/index.php\/wp-json\/wp\/v2\/categories?post=22"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/school.moongames.eu\/index.php\/wp-json\/wp\/v2\/tags?post=22"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}