3PP Upgrade

work

This is a record for my work to product 3pp upgrade.

Preparation

log on a host server to build docker image and test deployment:

  • log on a jump server first
  • ssh e-id@seroius01502

use xxxx/xxx-compiler(cdt2 image) to build frontend
use xxx/xxx-base(node image) to build backend

Compiler (frontend, cdt2) debugging

Run upgraded cdt2+node images as a container first:

1
2
$ docker run --rm -it -d --mount type=bind,src=/home/eshibij/cec-3pp-upgrade,dst=/usr/share sekidocker.rnd.ki.sw.ericsson.se/bolte/bcam/bcam-compiler:cdt_2.6.1-node_14.5 cat
e2849d6f818cb1ccacc1ce58c41d2695d0315b60b37f976c221397a997a856a2

[–rm](https://blog.csdn.net/nzjdsds/article/details/8hexo 1981732): clean up
-it: wait for container cmds to be executed
-d: run in background and return container id
–mount: https://blog.csdn.net/longlong6682/article/details/104730138
cat: seems like catting the container id

other refs:

1
2
3
4
5
$ docker ps
CONTAINER ID        IMAGE                                                                           COMMAND                  CREATED             STATUS              PORTS               NAMES
e2849d6f818c        .../bolte/bcam/bcam-compiler:cdt_2.6.1-node_14.5   "docker-entrypoint.s…"   2 seconds ago       Up 2 second
$ docker exec -it e284 bash
root@e2849d6f818c:/#
1
2
3
4
5
6
7
8
9
root@57b9f72a7d6a:/# USER=root
root@57b9f72a7d6a:/# sed -i "s/^\(.*\\"lamda-optimizer\":\).*/\1 \"^0.3.6\",/" /usr/local/lib/node_modules/@uisdk/cdt-build/package.json
root@57b9f72a7d6a:/# sed -i "s/^\(.*\"lamda\":\).*/\1 \"^0.2.3\",/" /usr/local/lib/node_modules/@uisdk/cdt-serve/package.json
root@57b9f72a7d6a:/# cd /usr/local/lib/node_modules/@uisdk/cdt-build/
root@57b9f72a7d6a:/# npm install --registry https://arm.sero.gic.ericsson.se/artifactory/api/npm/npm-remote/
root@57b9f72a7d6a:/# cd -
root@57b9f72a7d6a:/# cd /usr/local/lib/node_modules/@uisdk/cdt-serve/
root@57b9f72a7d6a:/# npm install --registry https://arm.sero.gic.ericsson.se/artifactory/api/npm/npm-remote/
root@57b9f72a7d6a:/# cd -
1
2
3
root@32f850a89d63:/# mkdir ~/.m2
root@32f850a89d63:/# cp docker/settings.xml ~/.m2
cp: cannot stat 'docker/settings.xml': No such file or directory

The docker/settings.xml does not exist because in real fast-test environment, all source code will be mounted to the container, here apparently no mounting step has been done. So I just make a docker directory and copy the settings file under it.

1
2
3
4
5
6
7
root@32f850a89d63:/# cat > docker/settings.xml << EOF
<settings xmlns="http://maven.apache.org/POM/4.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
          .....
> EOF
root@32f850a89d63:/# cp docker/settings.xml ~/.m2
1
2
root@32f850a89d63:/# cd coreservice/frontend
bash: cd: coreservice/frontend: No such file or directory

Same reason as before, so I mounted all source codes to container (better do it before start container)
If errors like permission denied, mkdir happens, try directly coping source code to container path. for example:

1
[home/eshibij/cec-3pp-upgrade]$ docker cp cec <container-id>:/mnt/share

issues

  1. lamda-optimizer

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    /usr/local/lib/node_modules/@uisdk/cdt-build/node_modules/lamda-optimizer/lamda-optimizer.js:25
        GLOBAL.define = lamda.define;
        ^
    ReferenceError: GLOBAL is not defined
        at module.exports (/usr/local/lib/node_modules/@uisdk/cdt-build/node_modules/lamda-optimizer/lamda-optimizer.js:25:5)
        at Object.executeLamda (/usr/local/lib/node_modules/@uisdk/cdt-build/tasks/compile.js:256:35)
        at Object.run (/usr/local/lib/node_modules/@uisdk/cdt-build/tasks/compile.js:115:14)
        at Runner.<anonymous> (/usr/local/lib/node_modules/@uisdk/cdt-build/config/runner.js:98:18)
        at Runner.emit (events.js:314:20)
        at Runner.<anonymous> (/usr/local/lib/node_modules/@uisdk/cdt-build/config/runner.js:72:18)
        at Runner.emit (events.js:314:20)
        at Runner.<anonymous> (/usr/local/lib/node_modules/@uisdk/cdt-build/config/runner.js:114:18)
        at Runner.emit (events.js:314:20)
        at Runner.<anonymous> (/usr/local/lib/node_modules/@uisdk/cdt-build/config/runner.js:106:18)

    Base (backend, node) debugging

    Container is already built up in above, so just enter the source code and run compile script. It turned out no error.
    Then come to the part of compile each service and run unit test. Some problems and solution during this proceed:

  2. target port is already in use

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    root@f7748a275a7a:/mnt/share/cellinfoservice# npm test
    > BCAM_Cell_Info_Handler@1.0.0 test /mnt/share/cellinfoservice
    > cross-env NODE_ENV=LOCAL mocha test --exit --timeout=10000
      1) Uncaught error outside test suite
      cell info server test
    (node:2739) [DEP0066] DeprecationWarning: OutgoingMessage.prototype._headers is deprecated
    (Use `node --trace-deprecation ...` to show where the warning was created)
        ✓ /
        ✓ post file (100ms)
        ✓ upload fail
        ✓ get files
        ✓ delete files
        ✓ import cells (93ms)
        ✓ import cells with iplanet (64ms)
      7 passing (440ms)
      1 failing
      1) Uncaught error outside test suite:
         Uncaught Error: listen EADDRINUSE: address already in use :::8082
          at Server.setupListenHandle [as _listen2] (net.js:1316:16)
          at listenInCluster (net.js:1364:12)
          at Server.listen (net.js:1450:7)
          ...

    solution: kill all node processes inside the container.

    • find all processes: docker exec <container-id> ps
      the different between docker exec ps and docker top <container-id> is that the previous one returns pid inside container, but the latter returns pid and ppid of these processes on host server. Both cmds are executed outside the target container.
    • enter container and kill all node processes using kill <pid>

    sample:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    [home/eshibij/cec-3pp-upgrade]$ docker exec f7748a275a7a ps
    PID TTY          TIME CMD
    1309 ?        00:00:00 node
    1331 ?        00:00:00 sh
    1332 ?        00:00:00 node
    1343 ?        00:00:00 node
    1365 ?        00:00:00 sh
    1366 ?        00:00:00 node
    1377 ?        00:00:00 node
    1399 ?        00:00:00 sh
    1400 ?        00:00:00 node
    2534 ?        00:00:00 node
    2546 ?        00:00:00 sh
    2547 ?        00:00:00 node
    2759 ?        00:00:00 ps
    [home/eshibij/cec-3pp-upgrade]$ docker exec -it f7748a275a7a bash
    root@f7748a275a7a:/# kill 1309
    root@f7748a275a7a:/# kill 1332
    root@f7748a275a7a:/# kill 1343
    ...
    [home/eshibij/cec-3pp-upgrade]$ docker exec f7748a275a7a ps
    PID TTY          TIME CMD
    2779 ?        00:00:00 ps

    Re-enter and run npm test to check unit test for each part:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    root@f7748a275a7a:/mnt/share/cellinfoservice# npm test
    > BCAM_Cell_Info_Handler@1.0.0 test /mnt/share/cellinfoservice
    > cross-env NODE_ENV=LOCAL mocha test --exit --timeout=10000
      cell info server test
    (node:2815) [DEP0066] DeprecationWarning: OutgoingMessage.prototype._headers is deprecated
    (Use `node --trace-deprecation ...` to show where the warning was created)
        ✓ /
        ✓ post file (88ms)
        ✓ upload fail
        ✓ get files
        ✓ delete files
        ✓ import cells (78ms)
        ✓ import cells with iplanet (53ms)
      7 passing (388ms)

  3. cross-env: Permission denied
    error code:

    1
    2
    3
    4
    5
    root@f7748a275a7a:/mnt/share/coreservice/backend# npm test
    > BCAM_Backend_Handler@1.0.0 test /mnt/share/coreservice/backend
    > cross-env NODE_ENV=LOCAL mocha test --exit --timeout=10000
    sh: 1: cross-env: Permission denied
    npm ERR! Test failed.  See above for more details

    solution: rebuild and run test again

    1
    $ npm rebuild

  4. local test failed due to "before all" hook

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    root@f7748a275a7a:/mnt/share/coreservice/backend# npm test
    > BCAM_Backend_Handler@1.0.0 test /mnt/share/coreservice/backend
    > cross-env NODE_ENV=LOCAL mocha test --exit --timeout=10000
    sequelize deprecated String based operators are now deprecated. Please use Symbol based operators for better security, read more at http://docs.sequelizejs.com/manual/tutorial/querying.html#operators node_modules/sequelize/lib/sequelize.js:245:13
      #model.js
        #Cell test()
          1) "before all" hook
      #ENMJobtest.js
        2) "before all" hook: create ENMJobTest Datas
        3) "after all" hook
      #ENodeBTest.js
        #ENodeB test()
          4) "before all" hook: create ENodeBTest Datas
      #model.js
        #MbsArea test()
          5) "before all" hook

    It happened due to no local db was found by the test procedures, create postgre db using image postgres:12.3-alpine with

    1
    $ docker run --name bcamdb -e POSTGRES_PASSWORD=bcam -p5432:5432 -d postgres:12.3-alpine

    Enter the db and get its local address:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    bash-5.0# ip a
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
    14468: eth0@if14469: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
        link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
        inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
           valid_lft forever preferred_lft forever

    Re-enter to backend container and ping the address to test connection:

    1
    2
    3
    4
    5
    6
    7
    root@f7748a275a7a:/# ping 172.17.0.2
    PING 172.17.0.2 (172.17.0.2) 56(84) bytes of data.
    64 bytes from 172.17.0.2: icmp_seq=1 ttl=64 time=0.067 ms
    64 bytes from 172.17.0.2: icmp_seq=2 ttl=64 time=0.050 ms
    ^C
      --- 172.17.0.2 ping statistics ---
    2 packets transmitted, 2 received, 0% packet loss, time 999ms

    re-write the backend env file application-default.yml and application-local.yml, retry and still failed.

    • check if connection is able to be built:
      1
      $ >/dev/tcp/172.17.0.2/5432
      no error throw out, so request to can be sent to db container
    • other debug:
      & means running in background, also can be added after npm test &, here I tried to do package capture but failed
      1
      2
      $ apt install tcpdump
      $ tcpdump -i any port 5432 &
      check process in running and kill process:
      1
      2
      3
      4
      $ ps -ef
      $ pkill npm
      $ pkill node
      $ pkill node npm
    • logon the default postgre: 
      1
      2
      $ su - postgres
      ddacfbb38c89:~$ psql
      It is the issue concerning with pg module: https://github.com/brianc/node-postgres/issues/2170
      update the version of pg to v8.0.3 or above solved this problem.

    updated solution
    use --net to attach your container directly to the host’s network interfaces.

    1
    $ docker run --name <container-name> -e POSTGRES_PASSWORD=postgres --net=container:<target-container-id> -d <image-name>:<tag>

    backend 3pp upgrade

    For upgrade part, firstly get to know https://docs.npmjs.com/cli/update and https://docs.npmjs.com/cli/install
    other refs:

Update step:

  • update package.json file with given versions

  • run npm install to update package in ./node_modules

  • run npm test for all UTs or add only before it to run a single test like：

    1
    /mnt/share/coreservice/backend $ npm test /test/CellTest.js -- -t "#Cell test()"

    Handle errors

    1. port is in use:
      1
      2
      1) Uncaught error outside test suite:
           Uncaught Error: listen EADDRINUSE: address already in use :::8083
      for case that netstat is not installed, use ss -lp instead to check port:
      1
      2
      3
      4
      root@f7748a275a7a:/mnt/share/enmservice# natstat -ano |grep 8083
      bash: natstat: command not found
      root@f7748a275a7a:/mnt/share/enmservice# ss -lp |grep 8083
      tcp    LISTEN     0      128    :::8083                 :::*                     users:(("node",pid=7662,fd=21))

      ss is auto-defined as “another utility to investigate sockets”. For more information you can use ss --help or man ss

    1
    $ kill 7662
    1. package connect-multiparty not compatible with node v14.
      ref: https://github.com/expressjs/connect-multiparty/issues/29
      change to lib multiparty or update node version to v14.6.0
  1. Error: EPERM: operation not permitted, unlink ‘C:\work\cec\saiservice\node_modules.staging\semver-9c480dbf\README.md’
    This usually occurs when the network connection is not stable. thx to https://blog.csdn.net/SilenceJude/article/details/101196261
    Solution:
    • delete node_modules folder in related package
    • npm install again
    • if not work, npm cache clean -f before installing